Metrics, logs and traces: a hands-on guide for engineers

Why observability matters, right now

Your application is running, requests are flowing through, and everything looks normal in the dashboard. Then the CEO reports that checkout is slow, but your CPU and memory metrics are clean. The problem exists somewhere in the request pipeline, but where? This is where metrics, logs and traces: a hands-on guide for engineers becomes essential. Understanding the three pillars of observability—metrics, logs, and traces—gives you the tools to answer different questions about your system's behaviour, from "is something wrong?" to "where exactly did this request spend five seconds?"

Most engineers rely on one or two of these signals in isolation and miss critical context. We'll walk through each pillar, show you what question it answers, and demonstrate how they work together to solve real production problems.

Compared — Each answers a different question

The confusion starts because metrics, logs, and traces all sound like "observability," but they're fundamentally different signals. Each one is optimized to answer a specific question.

Think of it like investigating a building after an alarm went off. Metrics are the motion sensors: they tell you something happened and where roughly. Logs are the security camera footage: they show you exactly what each room was doing. Traces are the path the suspect took: they connect all the rooms together and show you the sequence.

Metrics are cheap — Numbers over time, easy to alert on

A metric is a single number recorded at a point in time. Request count, response time percentile, database connection pool usage, error rate—these are all metrics. Because metrics are aggregated numbers (not full raw data), they're incredibly cheap to store and process. You can keep them forever.

Metrics answer the question: is something wrong, and how much? They're your early warning system.

Let's say you're running an e-commerce API. You expose a metric like this:

# TYPE http_request_duration_seconds histogram
# HELP http_request_duration_seconds Request latency in seconds
http_request_duration_seconds_bucket{le="0.1",endpoint="/api/products"} 450
http_request_duration_seconds_bucket{le="0.5",endpoint="/api/products"} 480
http_request_duration_seconds_bucket{le="1.0",endpoint="/api/products"} 495
http_request_duration_seconds_bucket{le="+Inf",endpoint="/api/products"} 500
http_request_duration_seconds_sum{endpoint="/api/products"} 150.5
http_request_duration_seconds_count{endpoint="/api/products"} 500

In 10 seconds, 500 requests hit /api/products. 95% of them finished in under 0.5 seconds. You can now write an alert:

- alert: SlowProductEndpoint
  expr: histogram_quantile(0.95, http_request_duration_seconds{endpoint="/api/products"}) > 1.0
  for: 5m
  annotations:
    summary: "95th percentile latency on /api/products exceeds 1 second"

This alert fires before users complain. Metrics let you trend over time, spot degradation, and correlate changes across your system. They're also stateless—the metric collector doesn't need to remember what happened last request.

The tradeoff: metrics are aggregated. You know how many requests were slow, but not which ones or why.

Logs are detailed — Rich context, expensive at scale

A log is a timestamped event. It contains text, structured fields, and context. Logs answer: what exactly happened?

When your metrics show that 5% of requests are slow, you need detail. This is where logs come in. A structured log entry might look like:

timestamp: 2025-01-15T14:32:51.234Z
level: WARN
trace_id: 6b4f8c9e-2d1a-4f7c-8a3b-5e6d9f2c1a8b
user_id: 42871
endpoint: /api/products/{id}
product_id: 9421
database_query: SELECT * FROM products WHERE id = ? LIMIT 1
db_duration_ms: 850
cache_hit: false
message: "Product lookup took 850ms, cache miss"

Now you know: this particular request missed the cache and had to hit the database. The database query took 850ms. You can search logs by trace_id, user, endpoint, or any field. Logs give you the story of what the system did.

The challenge with logs: they're expensive. At high volume, storing every log entry costs real money. A ride-hailing platform handling millions of requests per second can't log everything verbatim. Teams typically log warnings, errors, and sample a small percentage of successes.

Good practice: log at the right level and include context fields. Use structured logging (JSON), never raw text strings:

# Good
logger.info("checkout_processed", {
  order_id: "ord-9821",
  amount: 129.99,
  payment_processor: "stripe",
  duration_ms: 245,
  trace_id: "6b4f8c9e-2d1a-4f7c-8a3b-5e6d9f2c1a8b"
})

# Bad
logger.info("Order 9821 processed for $129.99 in 245ms")

The structured version is searchable and parseable. The unstructured version is noise.

Traces connect services — Follow one request end to end

A trace is a record of a single request's journey through your system. It shows which services touched it, how long each step took, and where it failed. Traces answer: where did the time go across services?

Imagine a request hitting your API gateway, then a product service, then a recommendation service, then a cache layer, then a database. The total round-trip was 2 seconds, but where? A trace breaks it down:

trace_id: 6b4f8c9e-2d1a-4f7c-8a3b-5e6d9f2c1a8b
spans:
  - span_id: api_gateway_001
    service: api-gateway
    operation: handle_request
    start_ms: 0
    duration_ms: 50
    status: success
  
  - span_id: product_service_001
    service: product-service
    operation: get_product_details
    start_ms: 50
    duration_ms: 600
    status: success
    child_spans: [cache_lookup, db_query]
  
  - span_id: cache_lookup
    service: redis
    operation: get
    start_ms: 50
    duration_ms: 5
    status: cache_miss
  
  - span_id: db_query
    service: postgres
    operation: SELECT
    start_ms: 55
    duration_ms: 540
    status: success
  
  - span_id: recommendation_service_001
    service: recommendation-service
    operation: get_recommendations
    start_ms: 650
    duration_ms: 800
    status: success
  
  - span_id: response_build
    service: api-gateway
    operation: build_response
    start_ms: 1450
    duration_ms: 50
    status: success

Now it's clear: the recommendation service took 800ms (40% of the total). The product database took 540ms (27%). Everything else was noise. You know exactly where to optimize.

Traces are expensive because you're storing detailed timing data for every span and every request. Most teams sample them—maybe 1% of all requests, or 100% of errors, or requests over a certain threshold. Tools like Jaeger or Datadog handle this sampling and correlation.

To use traces, instrument your code with a tracing library. For Node.js/NestJS, OpenTelemetry is the standard:

import { NodeTracerProvider } from '@opentelemetry/node';
import { JaegerExporter } from '@opentelemetry/exporter-jaeger';
import { SimpleSpanProcessor } from '@opentelemetry/sdk-trace-base';

const provider = new NodeTracerProvider();
const exporter = new JaegerExporter({ serviceName: 'product-service' });
provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.register();

Once instrumented, traces flow to your backend (Jaeger, Jaeger in this case) and you can query: "show me all traces where recommendation service took over 500ms."

How they work together

Real workflow: Your metrics alert fires. Response time on checkout is high. You jump into logs and filter by the same time window. You find several error entries with timeouts on the payment processor. You grab a trace_id from the logs and open it in your trace backend. The trace shows the payment API call hung for 8 seconds. Now you have direction: call the payment provider, check their status page, or implement a timeout fallback.

Without all three:

• Metrics alone: You know something is wrong but not what.
• Logs alone: You're searching blindly through thousands of entries.
• Traces alone: You only see a small sampled percentage of requests.

Together, they tell a complete story.

Common mistakes

• Not correlating with trace_id: Emit your trace_id in every log entry and metric label. This is how you connect the three pillars.
• Logging too much or too little: Log errors and warnings always. Log interesting success cases (e.g., cache miss, unusual latency) at INFO level. Sample the noise. Most teams log 10–20% of successful requests.
• Sampling traces randomly: Sample errors and slow requests at 100%, and fast/successful requests at 1–5%. This ensures you catch problems while keeping costs down.
• Forgetting to timestamp: Use UTC timestamps in all three signals. Timezone bugs waste hours.
• High cardinality labels: Don't use user_id as a metric label in Prometheus. Metrics with unbounded label values (like individual user IDs or order numbers) will explode your cardinality and crash your system. Use logs and traces for that granularity.

Key takeaways

• Metrics tell you is something wrong; logs tell you what happened; traces tell you where the time went.
• Metrics are cheap and always-on. Use them for alerting and trending.
• Logs are detailed but expensive. Use structured logging and include trace_id in every entry.
• Traces are very expensive. Sample wisely: 100% for errors, low percentage for noise.
• Link them with trace_id. A trace_id in your logs and trace backend is the glue that makes observability work.
• Each signal answers a different question. Use all three to move from "something is wrong" to "we know how to fix it."