Distributed tracing with Micrometer and OpenTelemetry

Why Distributed Tracing Matters in Production

When a request touches five different microservices and one of them fails, traditional logging leaves you hunting through log files across multiple servers. Distributed tracing with Micrometer and OpenTelemetry gives you a complete picture of that request's journey, showing you exactly where it slowed down, where it broke, and why. In this article, we'll walk through setting up distributed tracing with Micrometer and OpenTelemetry so your team can follow one request across every service and spot production issues before they impact users.

Add the Bridge — Micrometer-Tracing-Bridge-Otel Plus an OTLP Exporter

Spring Boot doesn't ship with tracing enabled by default. You need to add the Micrometer tracing bridge and an exporter that talks to your tracing backend. We typically use OpenTelemetry Protocol (OTLP) to send spans to a collector like Jaeger or Grafana Tempo, but you can also export directly.

Start by adding the dependencies to your pom.xml:

<dependency>
    <groupId>io.micrometer</groupId>
    <artifactId>micrometer-tracing-bridge-otel</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-otlp</artifactId>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk</artifactId>
</dependency>

If you're using Spring Boot 3.1+, you also want the Spring Boot starter for tracing:

<dependency>
    <groupId>org.springframework.boot</groupId>
    <artifactId>spring-boot-starter-micrometer-tracing-otlp</artifactId>
</dependency>

This starter bundles everything you need. Once it's on your classpath, Spring Boot's auto-configuration kicks in.

Now configure the OTLP exporter in your application.yml:

management:
  otlp:
    tracing:
      endpoint: http://localhost:4317
  tracing:
    sampling:
      probability: 0.1

The endpoint points to your OpenTelemetry collector. If you're running Jaeger locally with Docker, it listens on port 4317 by default. The probability: 0.1 means we'll sample 10% of traces—we'll refine this later.

At this point, Spring Boot automatically creates a global tracer and starts exporting spans. You don't need to wire anything manually. The magic happens in Spring's auto-configuration, which detects the bridge and exporter on your classpath and assembles them.

Trace IDs in Logs — Spring Adds traceId and spanId to the MDC Automatically

One of the biggest wins with Micrometer tracing is automatic MDC (Mapped Diagnostic Context) integration. Spring Boot automatically injects the current trace ID and span ID into SLF4J's MDC, so every log message includes them without you writing extra code.

Configure your logging pattern to include these values. In application.yml:

logging:
  pattern:
    level: "%5p"
    console: "%d{yyyy-MM-dd HH:mm:ss} - %msg%n%replace(%ex){'[\r\n]+', '\n'}%nopex [%thread] %logger{36} [%X{traceId}] [%X{spanId}] %5p"

The %X{traceId} and %X{spanId} placeholders pull values from the MDC. Now every log line includes the trace and span IDs:

2024-01-15 10:42:31 - Processing payment request
[main] com.example.PaymentService [a1b2c3d4e5f6] [x7y8z9w0] INFO

When you search your logs for trace ID a1b2c3d4e5f6, you'll see every log entry for that request across all services. This is where the real value emerges: you can correlate logs with traces in your UI without jumping between tools.

Spring also creates child spans automatically for common operations. A @Transactional method gets its own span. A REST call via RestClient gets a child span. Database queries get spans. You don't configure this—it's automatic instrumentation built into Spring integrations.

Propagation Works Out of the Box — RestClient, WebClient and Kafka Templates Forward Trace Headers

When service A calls service B, we need to propagate the trace context so both services' spans end up in the same trace. Micrometer handles this for you across Spring's client libraries.

If your service makes HTTP calls using RestClient:

@Service
public class OrderService {
    private final RestClient restClient;

    public OrderService(RestClient.Builder builder) {
        this.restClient = builder.baseUrl("http://payment-service").build();
    }

    public PaymentResult chargeOrder(Order order) {
        return restClient.post()
            .uri("/charge")
            .body(order)
            .retrieve()
            .body(PaymentResult.class);
    }
}

When this request goes out, Micrometer automatically adds trace headers like traceparent (W3C standard) and legacy headers. The receiving service extracts those headers and continues the trace. No configuration needed.

The same applies to WebClient for reactive calls and KafkaTemplate for async messaging. When you send a message to a Kafka topic, the trace context rides along in message headers. The consumer reads it and links the message processing back to the original trace.

This works because Spring's HTTP interceptors and Kafka interceptors are pre-instrumented. When you add the Micrometer bridge to your classpath, these interceptors automatically register themselves. We've seen teams waste weeks on "why isn't my trace context propagating?" only to realize they were using a custom HTTP client without proper instrumentation—stick to Spring's built-in clients and you're golden.

Sample Wisely — Sample 10% in Production and Always Keep Errors

Sampling is your safety valve. If you traced every request to every service, you'd drown in data and your bill would be astronomical. The default probability: 0.1 (10%) is a solid starting point for most teams, but sampling strategy depends on your traffic and budget.

The problem with uniform sampling is that rare errors might not get sampled. If an error occurs in 0.001% of requests and you sample 10%, you'll miss most of your errors. This is where adaptive sampling comes in.

Create a custom sampler that always captures errors:

@Configuration
public class TracingConfig {

    @Bean
    public Sampler sampler() {
        return new Sampler() {
            @Override
            public SamplingResult shouldSample(
                    Context parentContext,
                    long traceId,
                    String name,
                    SamplingDecision parentSamplingDecision) {
                
                // Always sample if there's an error flag
                if (parentContext != null) {
                    String samplingFlag = parentContext.get(AttributeKey.stringKey("error"));
                    if ("true".equals(samplingFlag)) {
                        return SamplingResult.recordAndSample();
                    }
                }
                
                // Otherwise use probabilistic sampling
                if ((traceId & Long.MAX_VALUE) % 100 < 10) {
                    return SamplingResult.recordAndSample();
                }
                return SamplingResult.drop();
            }

            @Override
            public String getDescription() {
                return "Error-aware sampler with 10% probabilistic fallback";
            }
        };
    }
}

In practice, you might use a library like OpenTelemetry's aws-xray-propagator or configure your backend (Jaeger, Tempo) to enforce tail-based sampling rules. With tail-based sampling, your collector decides which traces to keep after seeing the full trace, so you can say "keep all traces with errors" or "keep traces longer than 100ms" without losing data client-side.

In production, we've found that 10% uniform sampling plus error-based boosting catches most issues without breaking the bank. If you're running a high-volume platform, you might drop to 1-2%, but make sure your error sampling is aggressive.

Common Mistakes

• Forgetting to configure the exporter endpoint. Spans get created but silently dropped because the exporter has nowhere to send them. Always double-check your management.otlp.tracing.endpoint is reachable.
• Using custom HTTP clients without instrumentation. If you roll your own HTTP client or use an obscure library, it won't propagate trace headers. Stick to RestClient or WebClient where possible, or wrap custom clients with Micrometer manually.
• Sampling too aggressively in development. It's tempting to sample 100% locally, but this can hide sampling-related issues. Test your sampling strategy before pushing to production.
• Not including trace IDs in your logging format. Without %X{traceId} in your logs, you lose the ability to correlate logs with traces. This is a massive quality-of-life loss.
• Ignoring propagation for internal async work. If you spawn a background thread or scheduled task, the trace context is lost. Use ThreadLocal-aware patterns or pass the trace ID explicitly.

Key Takeaways

• Add micrometer-tracing-bridge-otel and an OTLP exporter to your classpath; Spring Boot auto-configures the rest.
• Include %X{traceId} and %X{spanId} in your logging format so every log line carries trace context.
• RestClient, WebClient, and KafkaTemplate propagate trace headers automatically—no extra plumbing required.
• Start with 10% sampling and add error-aware boosting to keep rare failures visible.
• Use a tracing backend like Jaeger or Grafana Tempo to visualize traces and correlate them with metrics and logs.