Tracing with OpenTelemetry

You've shipped an API endpoint that talks to your database, publishes a message to Kafka, and calls an external service—but some requests take 30 seconds when they should take 300ms. Without end-to-end visibility, you're flying blind. Tracing with OpenTelemetry gives you exactly that: a complete picture of a single request as it travels through your entire system, from the HTTP handler to the database query to the message broker and back.

I've been there. I once spent two hours debugging slow checkout requests on an e-commerce platform, only to discover the bottleneck wasn't our code at all—it was a third-party payment gateway timing out silently. With distributed tracing in place, I would have found it in minutes. This is why I make OpenTelemetry one of the first things I set up in production systems now.

Built into ASP.NET Core, HttpClient and EF Core

The good news: you don't need to add tracing from scratch. ASP.NET Core 6+ ships with built-in OpenTelemetry support that automatically instruments HTTP requests, database queries via Entity Framework Core, and outbound HTTP calls. All you need to do is enable it.

Start by adding the OpenTelemetry NuGet packages:

csharp

dotnet add package OpenTelemetry
dotnet add package OpenTelemetry.Exporter.Jaeger
dotnet add package OpenTelemetry.Instrumentation.AspNetCore
dotnet add package OpenTelemetry.Instrumentation.EntityFrameworkCore
dotnet add package OpenTelemetry.Instrumentation.Http

Then configure the tracer in your Program.cs:

csharp

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .WithTracing(tracingBuilder =>
    {
        tracingBuilder
            .AddAspNetCoreInstrumentation()
            .AddHttpClientInstrumentation()
            .AddEntityFrameworkCoreInstrumentation(options =>
            {
                options.SetDbStatementForText = true;
            })
            .AddJaegerExporter(options =>
            {
                options.AgentHost = "localhost";
                options.AgentPort = 6831;
            });
    });

var app = builder.Build();
app.UseRouting();
app.Run();

That's it. From this moment on, every HTTP request, every EF Core query, and every outbound HTTP call gets traced automatically. No code changes needed in your controllers or services.

What gets captured? The full span lifecycle: when the request starts, how long each database query takes, whether an HTTP call succeeded or timed out, and when the response leaves your API. Each span includes tags (like the HTTP status code or database table name) and events (like when an exception occurred).

Export to Jaeger, Grafana Tempo or Azure Monitor

Tracing data is useless if you can't see it. OpenTelemetry is exporter-agnostic, meaning you can send traces to any backend. I typically use three options depending on the environment:

Jaeger is the easiest for local development. It's a single Docker container that gives you a beautiful UI to search and visualize traces. I run it locally during development to find bottlenecks before they reach production.

Grafana Tempo is what I use for production on self-hosted infrastructure. It's designed to handle massive trace volume at low cost and integrates seamlessly with Prometheus and Grafana dashboards you may already have.

Azure Monitor (Application Insights) if you're on Azure. It's managed, requires minimal setup, and connects directly to your Azure resources.

Let me show you Jaeger setup since it's the fastest way to get started locally. First, spin up Jaeger:

csharp

docker run -d --name jaeger \
  -e COLLECTOR_ZIPKIN_HOST_PORT=:9411 \
  -p 6831:6831/udp \
  -p 16686:16686 \
  jaegertracing/all-in-one:latest

Then configure the exporter in your Program.cs (as shown above). Navigate to http://localhost:16686, and you'll see every trace from your API. Click on any trace and you'll see a waterfall diagram: the HTTP request at the top, then each database query and HTTP call indented below it with their exact duration.

For production, swap the Jaeger exporter for Tempo or Application Insights. The instrumentation code stays the same—you only change the exporter configuration.

Add custom spans with ActivitySource

The built-in instrumentation covers HTTP, database, and outbound calls. But what if you have custom business logic that matters to performance? Say you're processing a payment with a retry loop, or transforming a large dataset. You need to trace that too.

Enter ActivitySource. This is your API to create custom spans for any part of your code:

csharp

public class PaymentService
{
    private static readonly ActivitySource ActivitySource = new("MyApp.PaymentService");

    public async Task ProcessPaymentAsync(Order order)
    {
        using (var activity = ActivitySource.StartActivity("ProcessPayment"))
        {
            activity?.SetTag("order.id", order.Id);
            activity?.SetTag("order.amount", order.Total);

            try
            {
                activity?.AddEvent(new("CalculatingTax"));
                var tax = CalculateTax(order);

                activity?.AddEvent(new("ChargingCard"));
                var chargeResult = await ChargeCardAsync(order.PaymentMethod, order.Total + tax);

                activity?.SetTag("payment.status", chargeResult.Status);
                return new PaymentResult { Success = chargeResult.Success };
            }
            catch (Exception ex)
            {
                activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
                activity?.RecordException(ex);
                throw;
            }
        }
    }
}

Now when you view this trace in Jaeger, you'll see a "ProcessPayment" span with child spans for "CalculatingTax" and "ChargingCard". If the card charge takes 8 seconds while everything else takes 100ms, you've found your bottleneck instantly.

Don't overuse custom spans—they add overhead. I create spans only for work that could reasonably be slow: external API calls, CPU-intensive calculations, or anything involving retries or I/O. A simple property getter doesn't need a span.

The fastest way to find which hop is slow

This is the real magic of tracing. Imagine a request that follows this path:

HTTP request arrives at your API (5ms)
Authenticate user (1ms)
Load product from database (2ms)
Check inventory on a third-party API (12 seconds—timeout)
Calculate discount (1ms)
Save order to database (3ms)

Without tracing, your request took 12 seconds and you don't know why. You blame your infrastructure. You blame your database. You add more servers. None of it helps because the problem is a 12-second external API call.

With tracing, you open Jaeger, click on the slow request, and the waterfall diagram tells you exactly where time went. One span is clearly red or orange. One hop is obviously the villain. You either optimize the call, add caching, make it async, or set a timeout—and your problem is solved.

I've used this technique on ride-hailing platforms (finding that driver-match calculations were slow), on SaaS products (discovering a hidden N+1 query), and on trading systems (identifying that market data fetches weren't parallelized). Every time, tracing saved me hours of guesswork.

Common mistakes

Not exporting traces to production. Tracing locally is nice. Tracing production is essential. Set up Tempo or Application Insights before you ship. Local setup won't catch production-specific issues like network latency or high concurrency bottlenecks.
Creating too many custom spans. Each span has overhead. Span your business logic, not your utility functions. If your span is typically under 1ms, it's probably not worth the cost.
Ignoring trace retention policies. Traces are data-intensive. You can't keep every trace forever. Jaeger defaults to keeping the last 500 traces per service. In production, configure your backend to sample traces (e.g., 1 in 100 requests) or use more sophisticated sampling strategies.
Forgetting to configure SQL command logging in EF Core. By default, EF Core traces include the query type but not the actual SQL. Add SetDbStatementForText = true to your EntityFrameworkCoreInstrumentation options so you can see the actual SQL commands in traces. This is crucial for debugging slow queries.

Key takeaways

OpenTelemetry is built into ASP.NET Core 6+. Enable it with a few lines in Program.cs to automatically trace HTTP, database, and outbound HTTP calls.
Use Jaeger locally for development and Tempo or Application Insights in production. The instrumentation code stays the same; only the exporter changes.
Create custom spans with ActivitySource for your own business logic, but use them sparingly for work that could actually be slow.
Tracing turns vague performance complaints ("the API is slow") into specific answers ("the third-party API call takes 12 seconds").
Configure sampling and retention policies in production to manage trace volume and cost.

Tracing with OpenTelemetry: .NET lessons from production