Python logging for production: best practices explained

The problem: print() is not logging

We've all seen it. A production issue occurs, and someone SSH's into a server to tail application output, only to find a wall of print() statements mixed with error messages, timestamps that don't match server time, and no way to filter by severity or trace requests. This is the reality when teams skip proper logging: chaos masquerading as visibility.

Print statements are debugging tools. They disappear into the void when your app runs as a service, they don't integrate with log aggregation systems, and they can't be configured by environment. In contrast, Python logging for production: best practices explained means treating logs as structured data that feeds into monitoring pipelines, not as temporary debug noise.

The cost of getting this wrong compounds. When your SaaS application processes 10,000 requests a day and something goes wrong, you need to isolate the exact sequence of events for that one user's session. Print statements won't help. A properly structured logging system, with request IDs woven through the call stack, makes that investigation minutes instead of hours.

Use logging.getLogger(__name__) per module

The first rule: every module gets its own logger. This isn't optional. When you use logging.getLogger(__name__), Python creates a logger hierarchy that mirrors your package structure, and that hierarchy becomes invaluable when you need to adjust log levels or filter output by component.

# src/payment/processor.py
import logging

logger = logging.getLogger(__name__)

def process_transaction(user_id: int, amount: float) -> bool:
    logger.info(f"Processing transaction for user {user_id}, amount {amount}")
    try:
        result = charge_card(user_id, amount)
        logger.debug(f"Charge result: {result}")
        return result
    except PaymentGatewayError as e:
        logger.error(f"Payment gateway error: {e}", exc_info=True)
        raise

Notice the __name__ parameter. When this module runs, the logger's name becomes src.payment.processor. In production configuration, you can then set the payment module to DEBUG while keeping everything else at INFO. This granularity is impossible with global loggers or print statements.

The hierarchy also helps when third-party libraries go haywire. If requests library suddenly starts spamming DEBUG logs, you configure it:

logging.getLogger("urllib3").setLevel(logging.WARNING)

Three lines solve what would be 30 minutes of confusion with print statements.

Configure logging once at startup

Logging configuration should happen exactly once, in your application's entry point, before any other modules import anything. For a Django app, that's the wsgi.py or manage.py. For a FastAPI service, it's main.py. For a worker service, it's where your consumer loop starts.

The anti-pattern: configuring logging scattered across multiple modules, or calling basicConfig multiple times. This leads to duplicate handlers, lost configuration, and logs going nowhere.

Here's a solid pattern for a production service:

# src/main.py
import logging
import logging.config
import os
import json
from datetime import datetime

def configure_logging(environment: str) -> None:
    """Configure logging once at startup."""
    config = {
        "version": 1,
        "disable_existing_loggers": False,
        "formatters": {
            "json": {
                "()": "pythonjsonlogger.jsonlogger.JsonFormatter",
                "format": "%(timestamp)s %(level)s %(name)s %(message)s"
            },
            "simple": {
                "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s"
            }
        },
        "handlers": {
            "console": {
                "class": "logging.StreamHandler",
                "formatter": "json" if environment == "production" else "simple",
                "stream": "ext://sys.stdout"
            }
        },
        "root": {
            "level": "INFO" if environment == "production" else "DEBUG",
            "handlers": ["console"]
        },
        "loggers": {
            "sqlalchemy.engine": {"level": "WARNING"},
            "urllib3": {"level": "WARNING"}
        }
    }
    logging.config.dictConfig(config)

# In your app startup
if __name__ == "__main__":
    env = os.getenv("ENVIRONMENT", "development")
    configure_logging(env)
    # Now start your app
    app.run()

This approach gives you environment-aware configuration without code changes. Development gets readable console output, production gets JSON that your log aggregator can parse. Third-party loggers are muted at the source.

Emit JSON logs in production

Structured logging means JSON. Not "logstash format" or some home-rolled delimited format. JSON. Every log line is a valid JSON object that your log aggregation service (CloudWatch, Datadog, ELK, Loki) can parse without regex gymnastics.

The package python-json-logger makes this trivial:

pip install python-json-logger

Then in your logging config:

from pythonjsonlogger import jsonlogger

handler = logging.StreamHandler()
handler.setFormatter(jsonlogger.JsonFormatter())
logger.addHandler(handler)

logger.info("User created", extra={"user_id": 42, "email": "alice@example.com"})

Output:

{"timestamp": "2025-01-12T14:32:15Z", "level": "INFO", "name": "src.user", "message": "User created", "user_id": 42, "email": "alice@example.com"}

That's a single line that your aggregator can index by field. You can now query "all log lines where user_id=42" or "all errors in the payment module in the last hour" without string parsing.

One warning: don't over-log. Logging every variable at DEBUG level in production creates noise and cost (log storage isn't free). Log what matters: business events, errors with context, slow operations. Not internal loop iterations.

Add request IDs with contextvars

In a concurrent system (FastAPI, async workers, multi-threaded services), multiple requests execute interleaved. When you see an error in the logs, you need to know: which request was it? What happened before it in that request's lifecycle?

This is where request IDs and contextvars shine. A context variable is a thread-safe, async-safe namespace that travels with your execution context.

For a FastAPI service:

# src/logging_middleware.py
import logging
import uuid
from contextvars import ContextVar
from fastapi import Request
from pythonjsonlogger import jsonlogger

request_id_var: ContextVar[str] = ContextVar("request_id", default="")

class RequestIdFilter(logging.Filter):
    def filter(self, record):
        record.request_id = request_id_var.get()
        return True

@app.middleware("http")
async def add_request_id(request: Request, call_next):
    request_id = str(uuid.uuid4())
    request_id_var.set(request_id)
    response = await call_next(request)
    response.headers["X-Request-ID"] = request_id
    return response

# In logging config
handler = logging.StreamHandler()
handler.addFilter(RequestIdFilter())
handler.setFormatter(jsonlogger.JsonFormatter())
logger.addHandler(handler)

# Now every log line includes request_id
logger.info("Processing order")

Output:

{"timestamp": "2025-01-12T14:32:15Z", "level": "INFO", "name": "src.order", "message": "Processing order", "request_id": "a8f4e2c1-9d6b-4a2f-8e1c-3b7f2d4a6c9e"}

Now when a user reports an issue, you ask for the request ID from the response header, search your logs, and see the entire request's lifecycle in order. This is the difference between "the payment failed" and "the payment failed because we couldn't reach Stripe, which failed three times, then we retried and succeeded, but we logged the transaction twice."

For background workers or scheduled tasks where there's no HTTP request, generate a trace ID the same way and pass it through your message queue or execution context.

Common mistakes

Mistake 1: Logging sensitive data. Do not log passwords, credit card numbers, API keys, or auth tokens. Ever. A log aggregator breach or oversharing logs to a contractor exposes customer data. Sanitize:

def mask_sensitive(data: dict) -> dict:
    masked = data.copy()
    for key in ["password", "token", "credit_card"]:
        if key in masked:
            masked[key] = "***"
    return masked

logger.info("Login attempt", extra=mask_sensitive(request.data))

Mistake 2: Logging at the wrong level. INFO is for business events and operational milestones. DEBUG is for what you need when debugging. ERROR is for failures that need attention. Don't log every variable access as INFO.

Mistake 3: Catching and not logging exceptions. If you catch an exception, log it with exc_info=True to capture the stack trace, or use the exception() method:

# Good
except ValueError as e:
    logger.error(f"Invalid input: {e}", exc_info=True)

# Also good
except ValueError as e:
    logger.exception(f"Invalid input: {e}")

Mistake 4: Assuming logs go somewhere. In a container or serverless environment, stdout is captured by the orchestration layer (Docker, ECS, Lambda). But you must verify. Test in your actual deployment environment that logs appear in your aggregator.

Key takeaways

• Use logging.getLogger(__name__) in every module. This creates a logger hierarchy that lets you control log levels per component.
• Configure logging once at application startup, not scattered across files. Use dictConfig for environment-aware setup.
• Emit JSON logs in production. Use python-json-logger and let your aggregator parse structured fields.
• Attach request IDs via contextvars so you can trace a single request through concurrent execution.
• Log business events and errors with context. Don't log internal noise or sensitive data.
• Test logging in your actual deployment environment to confirm logs reach your aggregator.