Traces with Tempo or Jaeger: what developers should know

Introduction

After weeks of collecting metrics and correlating logs, you've built solid observability foundations. But when a customer reports a latency spike that spans five microservices, your dashboards show the aggregates—and your logs show individual events—yet neither tells you the complete picture of what happened to that single request. This is where distributed tracing comes in. Traces with Tempo or Jaeger: what developers should know is essential knowledge for anyone running production systems, because traces are the glue that connects a user action to every hop in your system. In this article, we'll explore how to choose between Tempo and Jaeger, design a tracing strategy that doesn't bankrupt you, and integrate traces with your existing metrics and logs.

Store traces cheaply in object storage

One reason teams hesitate to deploy distributed tracing is cost. A typical e-commerce platform generates millions of spans per day. If you try to store every trace in a traditional database, your storage bill becomes unsustainable.

Jaeger, the older of the two projects, was originally designed with Elasticsearch or Cassandra backends. These work, but they scale linearly with storage volume and require careful tuning. Many teams found themselves paying for retention they didn't need.

Tempo, built by Grafana Labs in 2020, took a different approach: store traces in object storage (S3, GCS, or MinIO) using a simple columnar format. Because object storage is cheap and infinite, Tempo trades query speed for cost-effectiveness. A single terabyte of traces might cost under $20/month on S3, whereas a comparable Elasticsearch cluster could cost thousands. Tempo also compresses traces aggressively, often achieving 10:1 compression ratios.

Here's how Tempo's storage layer works:

apiVersion: v1
kind: ConfigMap
metadata:
  name: tempo-config
data:
  tempo.yaml: |
    server:
      http_listen_port: 3100
    distributor:
      receivers:
        otlp:
          protocols:
            grpc:
              endpoint: 0.0.0.0:4317
            http:
              endpoint: 0.0.0.0:4318
    storage:
      trace:
        backend: s3
        s3:
          bucket: my-traces
          endpoint: s3.amazonaws.com
          access_key: ${AWS_ACCESS_KEY_ID}
          secret_key: ${AWS_SECRET_ACCESS_KEY}
    querier:
      frontend:
        compress_responses: true

In practice, Tempo stores traces in time-based blocks in object storage. The ingester buffers incoming spans, flushes them periodically (every 15 seconds to 5 minutes, configurable), and uploads complete blocks to S3. When you query a trace, Tempo downloads only the relevant blocks—not the entire dataset. This lazy loading keeps query latency reasonable even with massive retention windows.

When to choose Tempo: You have large trace volumes, long retention requirements (weeks or months), and a limited budget. You're willing to wait a few seconds for trace queries.

When to choose Jaeger: You need sub-second query performance, prefer an all-in-one deployment without external object storage, or need production-proven stability (Jaeger has been battle-tested since 2015).

Query by trace ID or attributes

Once traces are stored, you need to find them. This is where the developer experience diverges significantly.

The most common query pattern is by trace ID. When a user reports a problem, your support team copies the trace ID from a log or error message, pastes it into the tracing UI, and sees the full waterfall of every operation that request touched. This is the happy path.

But what if you don't have the trace ID? What if you only know that "a payment request failed around 2 PM in the us-west-2 region for a specific user"? This is where attribute queries matter.

Both Tempo and Jaeger support querying by span attributes (tags). For example:

service.name = "checkout" AND http.status_code = "500" AND user.id = "12345"

However, they differ in implementation.

Jaeger maintains indices in Elasticsearch or Cassandra. When you query by attribute, Jaeger hits the index, retrieves matching trace IDs, fetches the full traces, and returns them. This is fast because the index is kept hot in memory or on SSD. But it also means your storage backend must index every attribute, and cardinality explosions (e.g., unique user IDs, request IDs) can degrade performance.

Tempo uses a different strategy. It maintains a separate TraceQL index—a columnar store that lives alongside the block storage. Tempo doesn't index every attribute by default; instead, you declare which attributes matter for your queries. This is intentional: it keeps cardinality under control and storage costs predictable.

Here's an example Tempo query using TraceQL:

{
  resource.attributes.service.name = "payment-service" &&
  span.attributes.http.status_code = "500" &&
  span.attributes.error = true
}

In a real scenario, a payment service team might run this query to find all failed payment processing spans in the last hour, discover that 95% failed due to a downstream timeout, and identify the root cause as a DynamoDB throttle.

Practical consideration: Start with trace ID queries and add attribute queries only when you need them. Most incident resolution happens via trace ID lookups. Attribute queries are powerful but expensive if you're not disciplined about cardinality.

Link from logs and metrics

Traces don't exist in isolation. The true power of distributed tracing emerges when you link traces to your logs and metrics.

Imagine a metric dashboard shows that p99 latency spiked. You click through to a correlated log entry. The log says "Request ID: abc123def456". You copy that ID, paste it into Jaeger or Tempo, and see exactly which service was slow and why. This closed loop—metrics → logs → traces—is the gold standard of observability.

To enable this linking, every span and log entry must share a trace ID. When your application logs, it should include the current trace ID in the log context:

// Example in Node.js with OpenTelemetry
const { trace } = require('@opentelemetry/api');

const logger = {
  info: (message) => {
    const span = trace.getActiveSpan();
    const traceId = span ? span.spanContext().traceId : 'no-trace';
    console.log(JSON.stringify({
      level: 'info',
      message,
      trace_id: traceId,
      timestamp: new Date().toISOString()
    }));
  }
};

logger.info('Processing payment for user 5678');
// Output: {"level":"info","message":"Processing payment for user 5678","trace_id":"a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6","timestamp":"2024-01-15T10:32:45Z"}

Your logs storage (ELK, Loki, CloudWatch) should index the trace ID as a searchable field. Similarly, when emitting metrics (counters, histograms, gauges), you can include the trace ID as a label, though this is less common and can cause cardinality issues—be cautious.

Once linked, Grafana's UI offers direct navigation: click a trace ID link in a log, jump directly to Tempo or Jaeger, then navigate back to related logs. Many teams find this integration worth the implementation effort.

Sample to control cost

Even with cheap object storage, sampling is critical. A single request spanning ten services might generate 100 spans. If you're processing 100,000 requests per second, that's 10 million spans per second—terabytes per day. Sampling reduces this to a manageable volume.

Sampling strategies fall into three categories:

1. Head sampling: Decide at the entry point (the first span) whether to sample. This is simple and fast but can miss interesting patterns. If you sample 1% of requests, you'll miss 99% of errors unless errors are heavily weighted.
2. Tail sampling: Collect all spans initially, then decide retrospectively based on span attributes. If a span has status_code = 500, keep the entire trace. This is more accurate but requires buffering and processing overhead.
3. Dynamic sampling: Adjust sampling rates based on traffic, error rates, or latency percentiles. During high-traffic periods, lower the sample rate; during outages, raise it.

For most teams, head sampling with error oversampling strikes a balance. Sample 5% of successful requests but 100% of failed ones:

apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
data:
  config.yaml: |
    processors:
      probabilistic_sampler:
        sampling_percentage: 5
      error_sampler:
        status_code: "error"
        sampling_percentage: 100
    service:
      pipelines:
        traces:
          processors: [error_sampler, probabilistic_sampler]

With a probabilistic sampler at 5% and error oversampling at 100%, you capture representative slices of normal traffic plus all anomalies. A SaaS platform processing 100k requests/sec might retain traces from roughly 6k successful requests/sec (5%) plus all errors (~50/sec). That's manageable.

Calculation example: Assume 50 bytes per span (metadata only). 6,050 spans/sec × 50 bytes = 302.5 KB/sec = 26 GB/day. Over 90 days, that's 2.3 TB, costing roughly $50/month on S3. Without sampling, it would be 460 TB and $9,200/month.

Common mistakes

• Not sampling early: Teams deploy a tracing backend without sampling, and the bill shocks them after the first week. Start with aggressive sampling (1–5%) and dial it back based on retention and cost targets.
• Storing too many attributes: Instrumenting every local variable as a span attribute sounds helpful but explodes cardinality and query time. Stick to business-critical attributes: user ID, service name, error status, duration, HTTP method.
• Ignoring retention policies: Tempo and Jaeger both support retention windows. Set them explicitly. Old traces rarely matter; a 30-day retention is often sufficient and reduces storage cost by 60% compared to 90 days.
• Forgetting to link trace IDs to logs: Deploying tracing without correlating logs defeats half its purpose. Ensure your logging framework always includes the active trace ID.
• Choosing based on marketing, not requirements: Tempo is newer and trendy, but Jaeger is battle-tested. Evaluate your actual needs: Do you need sub-second queries? Can you accept 5-second latency for rare queries? Budget and performance requirements should drive the decision.

Key takeaways

• Tempo stores traces in object storage for cost efficiency (S3 ~$20/TB/month); Jaeger uses Elasticsearch or Cassandra for faster queries but higher cost.
• Always query by trace ID first. Attribute queries are powerful but should be added intentionally, not by default.
• Link traces to logs via shared trace IDs. This integration multiplies the value of your entire observability stack.
• Implement sampling immediately—head sampling with error oversampling works well for most platforms. Start at 5% for normal traffic and 100% for errors.
• Choose your backend based on query latency requirements and retention goals, not hype. Both Tempo and Jaeger are production-ready for different use cases.
• Set retention policies explicitly and monitor storage growth. Traces are high-volume; costs scale linearly without boundaries.