Tracing on AWS: concepts, trade-offs and production lessons

Tracing on AWS: concepts, trade-offs and production lessons

When a user clicks "buy now" in your e-commerce app, their request bounces across Lambda functions, hits DynamoDB, calls a payment API, and triggers an SQS message. If something breaks, where do you look? Without distributed tracing, you're flying blind. Tracing on AWS: concepts, trade-offs and production lessons gives you the visibility you need to follow requests across Lambda, ECS and APIs, pinpoint bottlenecks, and debug production issues in minutes instead of hours.

We've learned these patterns the hard way—through production incidents, cost overruns, and countless slow queries no one could explain. This guide covers X-Ray for service mapping, ADOT for flexible instrumentation, SDK-level tracing to surface database and API latency, and sampling strategies that keep costs reasonable.

Understanding distributed tracing on AWS

Distributed tracing solves a concrete problem: your microservices architecture makes single-box observability useless. A request that took 8 seconds involved five services, but which one was slow? Was it the DynamoDB query, the external API call, or network latency between containers?

A trace is a record of a request's journey. It's split into segments (one per service) and spans (one per operation—a database call, an HTTP request, a function invocation). Each span records:

• Operation name and duration
• Status (success or error)
• Metadata (user ID, order ID, exception details)
• Parent-child relationships to reconstruct the request flow

AWS provides two main tools: X-Ray (AWS-native) and ADOT (OpenTelemetry-based). Most teams use both—X-Ray for quick visualization, ADOT for rich instrumentation and flexibility.

X-Ray: Service maps and traces across AWS services

X-Ray is AWS's managed tracing service. It automatically captures calls between Lambda, ECS, API Gateway, and managed services like DynamoDB and S3. You enable it with a checkbox, and within seconds you get a service map showing which services communicate and how often requests fail.

To enable X-Ray in Lambda:

aws lambda update-function-configuration \
  --function-name my-function \
  --tracing-config Mode=Active

Also grant the Lambda execution role the AWSXRayDaemonWriteAccess policy. X-Ray then automatically wraps AWS SDK calls and logs them to the X-Ray daemon running on the Lambda runtime.

The service map view is X-Ray's killer feature. It shows you in real-time which services are calling which, success rates, and average latency. You spot a slow DynamoDB query affecting your checkout flow without reading a single log line.

For ECS, the X-Ray daemon runs as a sidecar container. Update your task definition:

{
  "name": "xray-daemon",
  "image": "public.ecr.aws/xray/aws-xray-daemon:latest",
  "portMappings": [
    {
      "containerPort": 2000,
      "protocol": "udp"
    }
  ]
}

Then your application container connects to localhost:2000 to send traces. X-Ray buffers them and sends them to the X-Ray API.

Reality check: X-Ray's automatic instrumentation is shallow. You see that DynamoDB was called, but not which items were queried or how many were scanned. You need ADOT for that depth.

ADOT: AWS Distro for OpenTelemetry sends traces to X-Ray or other backends

ADOT is AWS's supported distribution of OpenTelemetry, a vendor-neutral standard for collecting traces, metrics, and logs. It sits between your application and any backend—X-Ray, Datadog, New Relic, or your own system.

OpenTelemetry's architecture has three layers:

1. SDK — Your application instruments code and creates spans.
2. Collector — A sidecar or separate service receives spans, buffers them, and exports them.
3. Backend — Storage and visualization layer (X-Ray, Datadog, Jaeger, etc.).

The collector decouples your app from the backend, so you can switch from X-Ray to Datadog without touching application code. You also get built-in batching, retry logic, and sampling at the collector level.

For a Node.js Lambda function using OpenTelemetry SDK:

npm install @opentelemetry/api @opentelemetry/sdk-node \
  @opentelemetry/sdk-trace-node @opentelemetry/exporter-trace-otlp-http \
  @opentelemetry/instrumentation-express \
  @opentelemetry/instrumentation-http

Initialize tracing at your application's entry point:

const { NodeSDK } = require("@opentelemetry/sdk-node");
const { getNodeAutoInstrumentations } = require("@opentelemetry/auto-instrumentations-node");

const sdk = new NodeSDK({
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();

process.on("SIGTERM", () => sdk.shutdown());
process.on("SIGINT", () => sdk.shutdown());

That single block auto-instruments HTTP, Express, Node.js internals, and popular libraries. Spans are exported to your collector, which forwards them to X-Ray or another backend.

For .NET, the pattern is similar using the OpenTelemetry NuGet packages:

services.AddOpenTelemetry()
  .WithTracing(builder => builder
    .AddAspNetCoreInstrumentation()
    .AddHttpClientInstrumentation()
    .AddSqlClientInstrumentation()
    .AddOtlpExporter(opts => opts.Endpoint = new Uri("http://localhost:4317")));

The trade-off: ADOT adds a small overhead (CPU and memory) but gives you portability and fine-grained control. For most SaaS teams, the flexibility is worth it.

Instrument SDK calls: See time spent in DynamoDB, S3 and external APIs

Auto-instrumentation is powerful, but you often need custom spans to answer business questions. "Why did this order take 15 seconds?" requires tracing your business logic, not just framework calls.

Create manual spans using the OpenTelemetry API:

const tracer = require("@opentelemetry/api").trace.getTracer("order-service");

async function createOrder(userId, items) {
  const span = tracer.startSpan("createOrder", {
    attributes: {
      userId,
      itemCount: items.length,
    },
  });

  try {
    // Validate inventory
    const inventorySpan = tracer.startSpan("validateInventory", {
      parent: span,
    });
    await checkInventory(items);
    inventorySpan.end();

    // Write to DynamoDB
    const dbSpan = tracer.startSpan("putOrder", { parent: span });
    await dynamodb.putItem({
      TableName: "orders",
      Item: { orderId, userId, items, createdAt: Date.now() },
    }).promise();
    dbSpan.end();

    // Call payment API
    const paymentSpan = tracer.startSpan("chargePayment", { parent: span });
    await paymentService.charge(userId, items);
    paymentSpan.end();

    span.setStatus({ code: SpanStatusCode.OK });
  } catch (error) {
    span.recordException(error);
    span.setStatus({
      code: SpanStatusCode.ERROR,
      message: error.message,
    });
    throw error;
  } finally {
    span.end();
  }
}

This trace captures the full timeline: which step was slow, which threw an error, and how long each external call took. You can add attributes to tag spans by user, order ID, or A/B test variant.

For .NET, the pattern is nearly identical:

var tracer = TracerProvider.GetTracer("OrderService");

using (var span = tracer.StartSpan("CreateOrder"))
{
    span.SetAttribute("userId", userId);
    span.SetAttribute("itemCount", items.Count);

    try
    {
        await ValidateInventory(items);
        await DynamoDb.PutOrderAsync(order);
        await PaymentService.ChargeAsync(userId, total);
        span.SetStatus(Status.Ok);
    }
    catch (Exception ex)
    {
        span.RecordException(ex);
        span.SetStatus(new Status(StatusCode.Error, ex.Message));
        throw;
    }
}

Pro tip: instrument at the service boundary (before calling DynamoDB, S3, or external APIs). This shows you exactly where time is spent and surfaces timeouts, retries, or throttling issues.

Sample wisely: Trace a percentage of requests to control costs

Tracing every single request is expensive. X-Ray charges per million traces, and storing millions of traces daily adds up fast. The solution is sampling: trace 1%, 10%, or 100% of requests depending on volume and budget.

Sampling decisions happen in two places:

1. SDK-level (head sampling) — Your application decides whether to trace a request before doing work. This saves CPU since you don't create spans for unsampled requests.
2. Collector-level (tail sampling) — The collector receives all traces, then drops unsampled ones before export. This is slower but lets you sample based on latency, errors, or other runtime data.

For head sampling in OpenTelemetry, configure the sampler:

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter(),
  sampler: new TraceIdRatioBasedSampler(0.1), // 10% of requests
});

A sampler of 0.1 traces 10% of requests. 1.0 traces all (use only in dev). For head sampling, aim for 5–25% in production, depending on request volume and your budget. A busy SaaS with 10,000 requests per second can easily afford 1–5% sampling since that's still 100–500 traces per second.

Tail sampling is harder but more powerful. You might trace only requests that took over 1 second or returned an error. Configure the collector's tail sampler:

processors:
  tail_sampling:
    policies:
      - name: error-traces
        type: status_code
        status_code:
          status_codes: [ERROR]
      - name: slow-traces
        type: latency
        latency:
          threshold_ms: 1000
      - name: sample-remainder
        type: probabilistic
        probabilistic:
          sampling_percentage: 5

This traces all errors and slow requests, then samples 5% of the rest. Your SRE team gets visibility on production issues without the bill.

Cost reality: We've seen teams that traced 100% of requests and paid $50,000+ per month. Proper sampling cuts that to $2,000–5,000 for the same observability. Sample aggressively in production, higher in staging.

Common mistakes

• Not sampling at all, then shocked by the bill. Start with 10% and increase only if you have budget.
• Tracing only framework-level calls. Manual spans around business logic (order creation, payment processing) are where the insight lives.
• Forgetting to correlate traces with logs. Put the trace ID in log output so you can jump from a slow trace to the relevant logs. In Node.js, use middleware to inject it into every log.
• Using X-Ray alone without manual instrumentation. You'll see services calling each other but not why they're slow. Add OpenTelemetry SDK spans for real debugging power.
• Setting trace retention too high. Most teams keep traces for 7–30 days. Older traces rarely help, and retention is a cost lever.

Key takeaways

• Enable X-Ray on Lambda and ECS with a single setting; you get automatic service maps and error detection with near-zero code changes.
• Add OpenTelemetry SDK for fine-grained tracing—instrument business logic and service boundaries to surface where time actually goes.
• Manual spans around DynamoDB, S3, and external API calls show you bottlenecks and failures that service maps alone can't surface.
• Sampling is mandatory for cost control; start at 10–25% head sampling and use tail sampling to ensure errors and slow requests are always traced.
• Correlate traces with logs via trace ID injection; this lets you move fluidly from traces to detailed log context.
• The ADOT collector decouples your application from the backend, so switching from X-Ray to Datadog or Jaeger requires no application changes.
• In production, budget tracing spend like any other infrastructure cost; poorly configured sampling can easily exceed your observability bill.