Exemplars: everything engineers need to know in practice

The bridge between metrics and traces

When you spot a spike in your histogram—latency jumps, error rate climbs, throughput dips—the first question is always: which request caused this? Exemplars: everything engineers need to know in practice is that they solve this exact problem by embedding trace IDs directly into your metric samples. Instead of manually hunting through logs or traces, you click on a dot in Grafana and land in your trace backend. This single feature cuts investigation time from minutes to seconds.

In this article, we'll walk through how exemplars work end-to-end, from instrumentation to visualization, and show you how to wire them into your production systems.

Attach trace IDs to histogram samples

An exemplar is a trace ID paired with a metric observation. When your application records a histogram bucket—say, "request took 245ms"—it can also attach the trace ID of that specific request. Prometheus stores this pairing and makes it queryable through Grafana.

The critical requirement is the OpenMetrics format. Prometheus's classic text format doesn't support exemplars; you must emit metrics in OpenMetrics. Most modern instrumentation libraries handle this, but you need to explicitly enable it and pass trace context at instrument time.

Here's a practical example in Go with the OpenTelemetry SDK:

import (
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/metric"
	"go.opentelemetry.io/otel/trace"
)

func handleRequest(ctx context.Context, w http.ResponseWriter, r *http.Request) {
	start := time.Now()
	
	// Your request logic here
	processRequest(ctx)
	
	duration := time.Since(start).Seconds()
	
	// Get current span from context
	span := trace.SpanFromContext(ctx)
	traceID := span.SpanContext().TraceID().String()
	
	// Record histogram with trace ID as exemplar
	meter := otel.Meter("myapp")
	histogram, _ := meter.Float64Histogram("http.request.duration")
	
	histogram.Record(ctx, duration, metric.WithAttributes(
		attribute.String("http.method", r.Method),
		attribute.String("http.status", fmt.Sprintf("%d", 200)),
	))
}

The OpenTelemetry metric SDK automatically extracts the trace ID from the context and includes it in the exemplar when you record a measurement. When Prometheus scrapes your endpoint, it receives the metric value plus the attached trace ID.

In Node.js with the OpenTelemetry API:

const { trace, context, metrics } = require('@opentelemetry/api');

async function handleRequest(req, res) {
  const tracer = trace.getTracer('myapp');
  
  return tracer.startActiveSpan('http.request', async (span) => {
    const startTime = Date.now();
    
    // Your business logic
    await processRequest(req);
    
    const duration = (Date.now() - startTime) / 1000;
    const meter = metrics.getMeter('myapp');
    const histogram = meter.createHistogram('http.request.duration');
    
    // OpenTelemetry context is thread-local; exemplar is automatic
    histogram.record(duration, {
      'http.method': req.method,
      'http.status': res.statusCode,
    });
    
    span.end();
    res.send('OK');
  });
}

The key is that the instrumentation library reads the active span from context and bakes the trace ID into the metric emission. You don't need to manually pass it; it's automatic.

Prometheus stores exemplars in memory

When your app emits metrics in OpenMetrics format with exemplars, Prometheus scrapes them and stores the trace IDs in memory alongside the histogram buckets. By default, Prometheus keeps the most recent exemplar per histogram bucket.

Configure exemplar storage in your Prometheus config:

global:
  scrape_interval: 15s

# Enable exemplar storage (default is 100000)
metric_path: /metrics
scrape_configs:
  - job_name: 'myapp'
    static_configs:
      - targets: ['localhost:8080']

# Optional: increase exemplar storage size
# (number of exemplars kept in memory)
storage:
  exemplars:
    max_exemplars: 100000

One exemplar per bucket means if ten requests land in the 100ms histogram bucket, you'll see only one trace ID—the most recent one. This is intentional; keeping all exemplars for every bucket would consume unbounded memory. In practice, having the latest exemplar per bucket is usually enough to start your investigation.

Grafana shows exemplar dots on your chart

Once Prometheus is collecting exemplars, Grafana visualizes them as small dots overlaid on histogram panels. When you hover over or click an exemplar dot, Grafana displays the trace ID.

Set up a histogram panel in Grafana pointing to your Prometheus datasource:

- title: Request Duration
  targets:
    - expr: 'histogram_quantile(0.95, rate(http.request.duration_bucket[5m]))'
      datasource: 'Prometheus'
      legendFormat: 'p95'
  type: 'graph'
  options:
    legend:
      calcs: []
    tooltip:
      mode: 'multi'

Grafana will automatically render exemplar dots if your Prometheus version is 2.26+ and the metric is a histogram with exemplars. The dots appear as small circles or diamonds, usually in a contrasting color.

The exemplar panel in Grafana is context-aware. When you zoom into a time range or filter by labels, the exemplar dots update to show only those that match your query and timeframe. This is powerful for correlating a metric spike with actual traces.

Click through to Tempo or Jaeger

The real value of exemplars emerges when you configure Grafana to link to your trace backend. After installing and configuring Tempo or Jaeger as a datasource in Grafana, set up trace link routing:

For Tempo, configure it in Grafana's datasource settings:

apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
data:
  datasources.yaml: |
    apiVersion: 1
    datasources:
      - name: Prometheus
        type: prometheus
        url: http://prometheus:9090
        isDefault: true
        jsonData:
          exemplarTraceIdDestinations:
            - name: Tempo
              url: 'http://tempo:3100/api/traces/$${__value.raw}'
              datasourceUid: 'tempo-uid'
      - name: Tempo
        type: tempo
        url: http://tempo:3100
        uid: tempo-uid

Now when you click an exemplar dot in Grafana, it follows the link to Tempo, passing the trace ID as a parameter. Tempo retrieves and displays the full trace: spans, timing, attributes, events, and logs.

For Jaeger, the configuration is similar:

jsonData:
  exemplarTraceIdDestinations:
    - name: Jaeger
      url: 'http://jaeger:16686/trace/$${__value.raw}'
      datasourceUid: 'jaeger-uid'

The workflow is now seamless: spot a metric anomaly → click an exemplar dot → land in the trace backend and debug the exact request that caused the spike.

Requires OpenMetrics format

This is the hard requirement. Exemplars exist only in the OpenMetrics format, not in Prometheus's classic text format. If your instrumentation library or exporter doesn't support OpenMetrics, exemplars won't work.

Check your scrape endpoint to confirm OpenMetrics output. A valid OpenMetrics response includes the `# EOF` marker and exemplars in histogram buckets:

# HELP http_request_duration_seconds Request duration
# TYPE http_request_duration_seconds histogram
http_request_duration_seconds_bucket{le="0.005"} 10 # {trace_id="4bf92f3577b34da6a3ce929d0e0e4736"} 0.003 1609459200
http_request_duration_seconds_bucket{le="0.01"} 20
http_request_duration_seconds_bucket{le="+Inf"} 100
http_request_duration_seconds_sum 5.5
http_request_duration_seconds_count 100
# EOF

The `# {trace_id="..."} value timestamp` annotation is the exemplar. If you don't see these, your exporter isn't emitting OpenMetrics format.

Most cloud-native projects now default to OpenMetrics—Prometheus Java client, OpenTelemetry SDKs, and modern exporters all support it. But if you're instrumenting with older libraries or custom scrapers, verify format support before trying to use exemplars.

Common mistakes

• Forgetting to enable OpenMetrics format in your exporter: Your instrumentation library may support exemplars, but you must explicitly configure it to emit OpenMetrics. Many libraries default to classic format for backward compatibility.
• Not passing trace context to metric recording: If your tracer and meter aren't sharing the same context object, the span won't be available when recording metrics. Ensure your instrumentation library is configured to propagate context automatically.
• Expecting all exemplars to appear: Prometheus stores only one exemplar per histogram bucket, not one per request. If you have high traffic, most trace IDs will be overwritten. Use sampling in your trace backend to manage cost.
• Misconfiguring the trace link in Grafana: The URL template must match your trace backend's API. Test the link manually by constructing a URL with a known trace ID to verify it works before relying on it in production.
• Scrape interval too long: If your Prometheus scrape interval is 60s or more, exemplar dots will be sparse. Use 15–30s for better granularity when investigating incidents.

Key takeaways

• Exemplars embed trace IDs into metric samples, connecting metrics to traces in a single click.
• OpenMetrics format is mandatory; classic Prometheus text format does not support exemplars.
• Your instrumentation library must read active span context and attach trace IDs automatically during metric recording.
• Prometheus stores one exemplar per histogram bucket by default, keeping only the most recent observation.
• Configure trace link destinations in Grafana to route exemplar clicks to Tempo, Jaeger, or your trace backend.
• Exemplars are most valuable for high-latency or high-cardinality services where drilling into a single representative trace accelerates root cause analysis.