Observability for Python services: a step-by-step guide

Production failures often feel like they come out of nowhere—a customer reports slowness, your team scrambles through logs, and you realize you have no idea what the service was actually doing when it failed. Observability for Python services: a step-by-step guide walks you through the core practices that let you see inside your running system. By combining structured logs, metrics, traces, and error tracking, you shift from reactive firefighting to proactive understanding.

Why observability matters for Python backends

Python services power real workloads across e-commerce checkouts, CRM workflows, and trading systems. When they break, you need answers fast. Traditional logging—printing to stdout—doesn't scale. You lose request context, can't correlate events across services, and spend hours grep-ing through megabytes of logs.

Observability solves this by giving you three pillars: logs that tell a story, metrics that show trends, and traces that follow a request through your entire stack. In this guide, we'll build each piece step by step.

Structured logs with request IDs

Structured logging means emitting JSON instead of unformatted text. Every log entry becomes a machine-readable object with consistent fields. Add a request ID—a unique identifier per incoming request—and you can instantly find all activity related to one customer action.

Let's set up structured logging in a FastAPI service. We'll use python-json-logger and contextvars to automatically inject request IDs:

import logging
import uuid
from contextvars import ContextVar
from pythonjsonlogger import jsonlogger
from fastapi import FastAPI, Request
from fastapi.responses import JSONResponse

# Create a context variable for the request ID
request_id_var: ContextVar[str] = ContextVar('request_id', default='')

# Configure JSON logging
logger = logging.getLogger()
handler = logging.StreamHandler()
formatter = jsonlogger.JsonFormatter()
handler.setFormatter(formatter)
logger.addHandler(handler)
logger.setLevel(logging.INFO)

app = FastAPI()

@app.middleware("http")
async def add_request_id(request: Request, call_next):
    """Middleware to inject request ID into context."""
    req_id = request.headers.get('X-Request-ID', str(uuid.uuid4()))
    request_id_var.set(req_id)
    
    response = await call_next(request)
    response.headers['X-Request-ID'] = req_id
    return response

def get_logger():
    """Return a logger with request ID baked in."""
    logger_instance = logging.getLogger(__name__)
    # Patch the logger to add request_id automatically
    def log_with_context(level, msg, **kwargs):
        extra = kwargs.pop('extra', {})
        extra['request_id'] = request_id_var.get()
        logger_instance.log(level, msg, extra=extra, **kwargs)
    
    return logger_instance

@app.get("/checkout")
async def checkout(order_id: int):
    logger.info("Processing order", extra={"request_id": request_id_var.get(), "order_id": order_id})
    
    # Your business logic
    result = {"status": "success", "order_id": order_id}
    
    logger.info("Order processed", extra={"request_id": request_id_var.get(), "result": result})
    return result

Now every log line includes the request ID. In your log aggregation tool (ELK, Datadog, or similar), you can filter by request_id and see the entire lifecycle of a request—from entry to database calls to response. This cuts troubleshooting time from hours to minutes.

Prometheus metrics via Instrumentator

Logs tell you what happened. Metrics show you patterns and trends. Prometheus metrics are time-series data points—request latency, error rates, queue depth—that you graph and alert on.

For FastAPI, the instrumentator library automatically collects HTTP metrics without boilerplate:

from fastapi import FastAPI
from fastapi_instrumentator import Instrumentator
from fastapi_instrumentator.metrics import default_handler

app = FastAPI()

# Attach Instrumentator to your app
Instrumentator().instrument(app).expose(app)

@app.get("/api/process")
async def process():
    return {"status": "ok"}

This exposes metrics at /metrics in Prometheus format. You get latency percentiles, request counts by endpoint and method, and error rates automatically. Scrape the endpoint every 15 seconds and you can build dashboards showing:

• 95th percentile response time for your checkout endpoint
• Error rate spikes correlated with deployments
• Request volume trends across regions

For custom metrics—like order value or inventory levels—use the Prometheus client directly:

from prometheus_client import Counter, Histogram, Gauge

order_value = Histogram(
    'order_value_usd',
    'Order value in USD',
    buckets=[10, 50, 100, 500, 1000]
)

inventory_level = Gauge(
    'inventory_items_total',
    'Total items in stock',
    labelnames=['product_id']
)

@app.post("/orders")
async def create_order(amount: float):
    order_value.observe(amount)
    # Process order...
    return {"amount": amount}

@app.put("/inventory/{product_id}")
async def update_inventory(product_id: str, count: int):
    inventory_level.labels(product_id=product_id).set(count)
    return {"product_id": product_id, "count": count}

OpenTelemetry auto-instrumentation

Logs and metrics are powerful, but distributed tracing adds a crucial dimension. When a request hits your API, then calls a database, then publishes to Kafka, you need to see the entire path as one trace. OpenTelemetry gives you that without writing trace code yourself.

Install the auto-instrumentation packages:

pip install opentelemetry-api opentelemetry-sdk opentelemetry-exporter-jaeger
pip install opentelemetry-instrumentation-fastapi
pip install opentelemetry-instrumentation-sqlalchemy
pip install opentelemetry-instrumentation-requests

Then configure auto-instrumentation in your app startup:

from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.jaeger.thrift import JaegerExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from opentelemetry.instrumentation.requests import RequestsInstrumentor
from fastapi import FastAPI

app = FastAPI()

# Set up Jaeger exporter (sends traces to Jaeger backend)
jaeger_exporter = JaegerExporter(
    agent_host_name="localhost",
    agent_port=6831,
)

trace_provider = TracerProvider()
trace_provider.add_span_processor(BatchSpanProcessor(jaeger_exporter))

# Auto-instrument FastAPI, SQLAlchemy, and requests
FastAPIInstrumentor.instrument_app(app)
SQLAlchemyInstrumentor().instrument()
RequestsInstrumentor().instrument()

@app.get("/orders/{order_id}")
async def get_order(order_id: int):
    # Database query is automatically traced
    order = db.query(Order).filter(Order.id == order_id).first()
    
    # External HTTP call is automatically traced
    response = requests.get(f"https://payment-api/verify/{order_id}")
    
    return order
</code>

<p>Now every request generates a trace showing all database queries, HTTP calls, and their timing. You see not just that checkout was slow, but exactly which step—database lock? external API delay?—caused it. Jaeger's UI lets you visualize traces and spot bottlenecks instantly.</p>

<h2 id="sentry-for-error-tracking">Sentry for error tracking</h2>

<p>Errors happen. Without proper tracking, you might not know until a customer complains. Sentry captures exceptions, groups them by type and stack trace, and alerts your team.</p>

<p>Set it up in minutes:</p>

<pre><code class="language-python">import sentry_sdk
from sentry_sdk.integrations.fastapi import FastApiIntegration
from sentry_sdk.integrations.sqlalchemy import SqlalchemyIntegration
from fastapi import FastAPI

sentry_sdk.init(
    dsn="https://YOUR_KEY@YOUR_ORG.ingest.sentry.io/YOUR_PROJECT",
    integrations=[
        FastApiIntegration(),
        SqlalchemyIntegration(),
    ],
    traces_sample_rate=0.1,  # Capture 10% of transactions
    environment="production",
)

app = FastAPI()

@app.get("/process")
async def process():
    # This exception is automatically sent to Sentry
    result = risky_operation()
    return result

@app.exception_handler(Exception)
async def exception_handler(request, exc):
    # Sentry captures this automatically
    raise exc

Sentry groups errors by fingerprint (stack trace), so duplicate errors don't spam you. You see error frequency, affected users, and the exact code line. Integrate with Slack or PagerDuty to alert on critical errors in real time.

Attach context to help debugging:

sentry_sdk.set_context("order", {
    "order_id": 12345,
    "customer_id": 67890,
    "status": "processing"
})

sentry_sdk.set_tag("service", "checkout")
sentry_sdk.set_tag("region", "us-east-1")

Tying it together

In a real service, these four pieces work together. A request comes in, gets a unique ID (logs), hits your endpoint (metrics count it), calls a database (trace shows latency), and if it fails, Sentry captures it with full context. Your dashboard shows throughput and latency. Your logs show what each request did. Your traces show where time was spent. Sentry alerts you to errors before users do.

Deploy this stack incrementally. Start with structured logging and Prometheus metrics—they're low-overhead and give immediate value. Add OpenTelemetry tracing next for distributed visibility. Finally, integrate Sentry for error handling. Each piece works standalone but together they form a complete picture of your system's health.

Common mistakes

• Logging too much: Every debug statement at info level fills your log storage. Use debug level for noisy details and reserve info/warn/error for actionable events. Sample high-volume logs to keep storage costs down.
• Ignoring cardinality: In Prometheus, unique label combinations create new time series. If you label metrics by customer ID or request ID, you'll create millions of series and crash your monitoring. Use bounded labels (status codes, endpoint paths) instead.
• No alerting rules: Metrics without alerts are just pretty graphs. Define alert thresholds for error rate spikes, latency percentiles, and queue depth. Test them before production.
• Forgetting to correlate: Request IDs in logs, trace IDs in spans, and fingerprints in Sentry are useless if they don't link. Make sure your observability platform (or manual linking) connects all three.
• Sampling traces indiscriminately: Sampling 1% of traces saves money but loses rare errors. Use tail-based sampling (capture traces with errors, slow requests, or rare paths) instead of head-based sampling (random capture at ingestion).

Key takeaways

• Structured logging with request IDs lets you follow a single request through your entire system and correlate activity across services.
• Prometheus metrics via Instrumentator provide automatic HTTP observability—latency, errors, request counts—without code changes.
• OpenTelemetry auto-instrumentation traces requests end-to-end, showing exactly where time is spent in database calls, external APIs, and compute.
• Sentry captures unhandled exceptions, groups them intelligently, and alerts your team before customers discover bugs.
• Combine all four—logs, metrics, traces, and errors—for complete observability. Each solves a different question: what happened, what's the trend, where's the bottleneck, and what broke.
• Deploy incrementally, starting with low-overhead signals (structured logs, Prometheus) before adding higher-overhead tracing.