OpenTelemetry tracing in FastAPI: lessons from production

Why tracing matters in production

When a request takes 5 seconds instead of 500ms, which database query is the culprit? When a payment flow fails for 2% of users but passes in staging, where exactly does it break? These are questions that logs alone cannot answer reliably. OpenTelemetry tracing in FastAPI: lessons from production teaches us that distributed tracing is not optional once you move beyond a single service—it becomes your window into what's really happening when systems interact.

We've learned this the hard way. A seemingly simple request would traverse our FastAPI service, call three different microservices via HTTP, hit PostgreSQL twice, and cache results in Redis. When latency spiked, we had no coherent picture of where the time went. Adding structured tracing changed everything: we could see each span, each service boundary, each database call, all correlated by a single trace ID. This article captures what we wish we'd known from day one.

Install opentelemetry-instrumentation-fastapi and an OTLP exporter

Start with the right packages. The OpenTelemetry Python ecosystem is modular—you pick what you need and compose it together. For a FastAPI service that also makes outbound HTTP calls and runs database queries, you'll need several packages working in tandem.

pip install opentelemetry-api opentelemetry-sdk \
  opentelemetry-instrumentation-fastapi \
  opentelemetry-instrumentation-sqlalchemy \
  opentelemetry-instrumentation-httpx \
  opentelemetry-instrumentation-requests \
  opentelemetry-exporter-otlp-proto-http

Why these specific packages? opentelemetry-api and opentelemetry-sdk provide the core tracing infrastructure. The instrumentation packages hook into FastAPI, SQLAlchemy, and HTTP clients to automatically capture spans. The OTLP exporter sends traces over HTTP to your backend (Jaeger, Grafana Tempo, Datadog, or a similar service).

We typically also install opentelemetry-exporter-jaeger for local development, where we can run Jaeger in Docker and see traces immediately without cloud setup:

docker run -d --name jaeger \
  -e COLLECTOR_OTLP_ENABLED=true \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 16686:16686 \
  jaegertracing/all-in-one:latest

Port 16686 is the Jaeger UI. Port 4317 is the gRPC OTLP endpoint; 4318 is HTTP. We'll use 4318 for simplicity.

Auto-instrument FastAPI, httpx, and SQLAlchemy in a few lines

Once packages are installed, instrumentation is remarkably simple. You call a setup function for each component, and from that moment forward, those libraries emit spans automatically. This is one of OpenTelemetry's best features: you get observability without rewriting your business logic.

Here's a minimal setup in your FastAPI application startup:

from fastapi import FastAPI
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter
from opentelemetry.instrumentation.fastapi import FastAPIInstrumentor
from opentelemetry.instrumentation.sqlalchemy import SQLAlchemyInstrumentor
from opentelemetry.instrumentation.httpx import HTTPXClientInstrumentor
from opentelemetry import trace

app = FastAPI()

# Create an OTLP exporter pointing to Jaeger
otlp_exporter = OTLPSpanExporter(
    endpoint="http://localhost:4318/v1/traces"
)

# Create a tracer provider and add the exporter
trace_provider = TracerProvider()
trace_provider.add_span_processor(BatchSpanProcessor(otlp_exporter))
trace.set_tracer_provider(trace_provider)

# Auto-instrument the libraries
FastAPIInstrumentor.instrument_app(app)
SQLAlchemyInstrumentor().instrument()
HTTPXClientInstrumentor().instrument()

That's genuinely all you need to start seeing traces. Every request to your FastAPI endpoints will now generate a span. Every SQL query will be captured. Every outbound HTTP call (via httpx) will create a child span linked to the request.

In production, replace the localhost endpoint with your actual collector. Many teams use environment variables:

import os

otlp_endpoint = os.getenv(
    "OTEL_EXPORTER_OTLP_ENDPOINT",
    "http://localhost:4318/v1/traces"
)
otlp_exporter = OTLPSpanExporter(endpoint=otlp_endpoint)

The BatchSpanProcessor batches spans and sends them asynchronously so that tracing doesn't block your request handlers. This is critical for production; synchronous span export would tank your latency.

Add custom spans for business logic

Auto-instrumentation gets you 80% of the way there, but you often need to trace your own business logic. Not every operation is a database call or HTTP request. Payment validation, fraud detection, cache lookups, or business rule evaluation might be invisible to the standard instruments.

Here's how to add a custom span around an important operation:

from opentelemetry import trace

tracer = trace.get_tracer(__name__)

@app.post("/orders")
async def create_order(order_data: OrderRequest):
    with tracer.start_as_current_span("validate_inventory") as span:
        span.set_attribute("order_id", order_data.id)
        span.set_attribute("item_count", len(order_data.items))
        
        # Your validation logic here
        for item in order_data.items:
            if not await check_stock(item.sku, item.quantity):
                span.set_attribute("validation_failed", True)
                raise OutOfStockError(item.sku)
        
        span.set_attribute("validation_passed", True)
    
    # Continue with order processing

The span name ("validate_inventory") appears in the trace timeline. Attributes let you attach business context—order IDs, quantities, validation results—so you can filter and search traces later. When something goes wrong, you won't just see "this step took 2 seconds"; you'll see exactly which order failed and why.

For longer-running tasks or operations that involve external dependencies, nest spans:

with tracer.start_as_current_span("process_payment") as parent_span:
    parent_span.set_attribute("amount_cents", amount_cents)
    
    with tracer.start_as_current_span("call_payment_gateway") as child_span:
        child_span.set_attribute("gateway", "stripe")
        response = await stripe_client.charge(amount_cents)
    
    if response.success:
        with tracer.start_as_current_span("update_order_status") as child_span:
            child_span.set_attribute("new_status", "paid")
            await db.update_order_status(order_id, "paid")

The trace UI will show this as a parent span with two child spans. You can see the timeline of each step and identify bottlenecks immediately.

Export traces to Jaeger, Tempo, or a managed backend

In development, Jaeger is ideal. For production, many teams use Grafana Tempo (open source, cost-effective) or managed services like Datadog, New Relic, or Honeycomb. The OTLP protocol is a standard, so switching backends is straightforward.

If you're running Tempo in Kubernetes:

otlp_exporter = OTLPSpanExporter(
    endpoint="http://tempo:4318/v1/traces"
)

For Datadog (which also accepts OTLP):

otlp_exporter = OTLPSpanExporter(
    endpoint="https://api.datadoghq.com/v1/traces",
    headers={
        "DD-API-KEY": os.getenv("DD_API_KEY")
    }
)

One production lesson: always use sampling in high-traffic services. Exporting every single span from a service handling 10,000 requests per second is wasteful and expensive. OpenTelemetry provides sampling policies:

from opentelemetry.sdk.trace.sampling import TraceIdRatioBased

sampler = TraceIdRatioBased(0.1)  # Sample 10% of traces
trace_provider = TracerProvider(sampler=sampler)
trace_provider.add_span_processor(BatchSpanProcessor(otlp_exporter))
trace.set_tracer_provider(trace_provider)

Be careful: sampling biases your latency metrics. If you sample 10%, you might miss the slowest outlier traces. Many teams use probabilistic sampling in normal conditions but disable it (or set it to 100%) for errors or high-latency requests using a more sophisticated sampler.

Common mistakes

Using synchronous span export. If you use SimpleSpanProcessor instead of BatchSpanProcessor, every span blocks until it's exported. Your API will feel sluggish. Always batch in production.

Forgetting to close spans. If you use start_span() without a context manager and forget to end(), spans leak memory. Always use with tracer.start_as_current_span(...).

Over-instrumenting with high cardinality attributes. Don't attach user IDs, request paths with variables, or other high-cardinality fields as span attributes without care. They'll explode your cardinality and make the backend too large. Use low-cardinality attributes (service name, endpoint, status, error type) and store specific data in logs.

No sampling strategy. Sending every trace to the backend works fine at 100 RPS but becomes impossibly expensive at 100,000 RPS. Plan your sampling from the start.

Ignoring span context propagation. If you call another service and don't propagate the trace ID, you'll lose visibility across service boundaries. OpenTelemetry's instrumentation does this automatically for HTTP headers, but if you use custom communication (gRPC, Kafka, custom headers), you must propagate manually.

Key takeaways

• Start with opentelemetry-instrumentation-fastapi, an OTLP exporter, and auto-instrumentation. You'll get request, database, and HTTP tracing almost instantly.
• Wrap business-critical operations in custom spans using tracer.start_as_current_span(). Attach low-cardinality attributes so you can search and correlate traces later.
• Use BatchSpanProcessor for production to avoid blocking your request handlers. Always implement sampling to manage backend costs and cardinality.
• Export to Jaeger in development; move to Tempo, Datadog, or similar in production. The OTLP protocol makes switching painless.
• Propagate trace context across service boundaries automatically (via HTTP instrumentation) or manually (via custom transports). A broken trace chain defeats the purpose.
• Invest time early in trace naming and attribute design. Good naming conventions (e.g., validate_inventory, call_payment_gateway) make traces readable under stress.