Observability: logs, metrics and traces, explained simply

Observability: logs, metrics and traces

You're on-call at 2 AM. Alerts are firing, but you have no idea why. You check the dashboard—CPU is high, error rate spiked—but that doesn't tell you which request is broken or what actually went wrong inside your code. This is where observability: logs, metrics and traces becomes your lifeline. These three signals answer different questions about your system, and only together do they paint a complete picture.

Three signals — each answers a different question

In production systems, you'll hear people conflate logging with observability, or treat metrics as optional. That's backwards. Each signal exists because it answers a fundamentally different question.

Metrics: Is something wrong?

Metrics are numeric measurements over time. They're aggregated, they're cheap to store, and they're your early warning system. When you ask "is something wrong?", metrics answer first.

A metric tells you: request rate is 500 req/s, 99th percentile latency is 250ms, error rate is 0.5%, CPU is 72%, memory is 4.2GB. You can set thresholds and alert. You can see trends over hours or days. Metrics are the dashboard you stare at when the pager goes off.

I typically expose three foundational metrics per service:

• Request rate – absolute throughput
• Error rate – percentage of requests that failed
• Latency percentiles – p50, p95, p99 response times

These are known as RED metrics (Rate, Errors, Duration). In a real system, you'd also track resource metrics (CPU, memory, disk) and business metrics (orders placed, revenue). But RED is your minimum viable observability.

Here's a simple Node.js example using Prometheus client:

import { register, Counter, Histogram, Gauge } from 'prom-client';

const httpRequestDuration = new Histogram({
  name: 'http_request_duration_seconds',
  help: 'Duration of HTTP requests in seconds',
  labelNames: ['method', 'route', 'status_code'],
  buckets: [0.01, 0.05, 0.1, 0.5, 1, 2, 5],
});

const httpRequestsTotal = new Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status_code'],
});

app.use((req, res, next) => {
  const start = Date.now();
  res.on('finish', () => {
    const duration = (Date.now() - start) / 1000;
    httpRequestDuration.observe(
      {
        method: req.method,
        route: req.route?.path || 'unknown',
        status_code: res.statusCode,
      },
      duration
    );
    httpRequestsTotal.inc({
      method: req.method,
      route: req.route?.path || 'unknown',
      status_code: res.statusCode,
    });
  });
  next();
});

Metrics tell you that something is wrong. They don't tell you what or why.

Traces: Where is it slow or failing?

A trace is a record of a single request's journey through your system. It shows you every service the request touched, how long each step took, and where it failed. When you ask "where is it slow or failing?", traces answer.

A trace contains spans. Each span is a unit of work—a database query, an API call, a business logic operation. A span has a start time, a duration, and metadata (service name, operation name, tags). Spans form a tree: a root span (the user's HTTP request) with child spans (calls to other services, database queries).

If you have 100,000 requests per second, you can't store every trace. You sample them. In dev or staging, sample at 100%. In production, sample at 1 in 100. When an error occurs, sample at 100% (error-driven sampling). This keeps costs reasonable while keeping visibility where it matters.

Here's how you'd instrument a simple transaction using OpenTelemetry:

import { trace, context, SpanStatusCode } from '@opentelemetry/api';

const tracer = trace.getTracer('myapp');

async function processOrder(orderId: string) {
  const span = tracer.startSpan('processOrder');
  
  try {
    await context.with(trace.setSpan(context.active(), span), async () => {
      const fetchSpan = tracer.startSpan('fetchOrderDetails');
      const order = await db.orders.findById(orderId);
      fetchSpan.end();

      const paymentSpan = tracer.startSpan('processPayment');
      paymentSpan.setAttributes({
        'payment.amount': order.total,
        'payment.currency': order.currency,
      });
      await paymentService.charge(order.total);
      paymentSpan.end();

      const inventorySpan = tracer.startSpan('updateInventory');
      await inventory.reserve(order.items);
      inventorySpan.end();
    });
    
    span.setStatus({ code: SpanStatusCode.OK });
  } catch (error) {
    span.recordException(error);
    span.setStatus({ code: SpanStatusCode.ERROR });
    throw error;
  } finally {
    span.end();
  }
}

A distributed trace gives you the timeline. You see that payment processing took 850ms when it usually takes 50ms. You see that the inventory service timed out. You see that the database query returned 50k rows instead of 50. Traces are where you do root cause analysis.

Logs: Why did it happen?

Logs are human-readable (or JSON-readable) text records of discrete events. When you ask "why did it happen?", logs answer. Metrics and traces show you the skeleton; logs show you the tissue and blood.

A log line might be: "User 12345 failed login: incorrect password" or "Inventory service returned HTTP 503" or "Database connection pool exhausted, waiting 2.3s". Logs capture the detail that metrics and traces can't convey. They're also the most verbose signal—you can't afford to log everything in production, so you log strategically.

I follow this rule: log what you need to understand why something failed or behaved unexpectedly. Log errors, warnings, and interesting business events. Don't log every single operation—that's what traces are for.

import pino from 'pino';

const logger = pino({
  transport: {
    target: 'pino-pretty',
  },
  level: process.env.LOG_LEVEL || 'info',
});

async function authenticateUser(username: string, password: string) {
  try {
    const user = await db.users.findByUsername(username);
    if (!user) {
      logger.warn({ username }, 'User not found');
      throw new Error('Invalid credentials');
    }

    const isValid = await bcrypt.compare(password, user.passwordHash);
    if (!isValid) {
      logger.warn({ userId: user.id, username }, 'Password mismatch');
      throw new Error('Invalid credentials');
    }

    logger.info({ userId: user.id }, 'User authenticated');
    return user;
  } catch (error) {
    logger.error(
      { error: error.message, username, stack: error.stack },
      'Authentication failed'
    );
    throw error;
  }
}

Notice the structured logging: each log line includes context (userId, username) as key-value pairs. This makes logs searchable and machine-readable, not just human-readable.

Correlate them — the trace ID is the golden thread

You now have three signals. But they're useless if they're disconnected. The key is correlation: every log line and every metric needs to know which trace it belongs to.

The solution: a trace ID. When a request enters your system, generate a unique ID (UUID). Pass it through every service, database call, and log statement. At the end, you can query: "show me all logs and traces for trace ID abc-123". Suddenly, all three signals become one coherent story.

In OpenTelemetry, the trace ID is generated automatically. But you need to inject it into your logs manually:

import { trace } from '@opentelemetry/api';

function getTraceId() {
  const activeSpan = trace.getActiveSpan();
  if (!activeSpan) return null;
  return activeSpan.spanContext().traceId;
}

app.use((req, res, next) => {
  const traceId = getTraceId();
  logger.info({ traceId, method: req.method, url: req.url }, 'Request started');
  res.on('finish', () => {
    logger.info(
      { traceId, statusCode: res.statusCode },
      'Request completed'
    );
  });
  next();
});

Now when you look at a metric alert, you click into a trace. The trace shows you which service is slow. The same trace ID is on every log line in that service. You can jump from dashboard → trace → logs → source code. This is observability.

Instrument with standards — OpenTelemetry avoids vendor lock-in

Ten years ago, every vendor had their own instrumentation format. New Relic, DataDog, Dynatrace—all proprietary. This meant you couldn't easily switch tools. OpenTelemetry changed that.

OpenTelemetry (OTel) is a vendor-neutral standard for collecting metrics, traces, and logs. You instrument your code once with OTel. Then you can export to any backend: Jaeger, Prometheus, Grafana Loki, Datadog, New Relic, Honeycomb—your choice, with no code changes.

OTel has been adopted by almost every major cloud platform and observability tool. It's become the industry standard. Using it means you're future-proofing your observability.

Here's a minimal setup for Node.js with OpenTelemetry exporting to a local Jaeger instance:

import { NodeTracerProvider } from '@opentelemetry/node';
import { JaegerExporter } from '@opentelemetry/exporter-jaeger-http';
import { BatchSpanProcessor } from '@opentelemetry/sdk-trace-base';
import { registerInstrumentations } from '@opentelemetry/auto-instrumentations-node';

const jaegerExporter = new JaegerExporter({
  endpoint: 'http://localhost:14268/api/traces',
});

const tracerProvider = new NodeTracerProvider();
tracerProvider.addSpanProcessor(new BatchSpanProcessor(jaegerExporter));
tracerProvider.register();

registerInstrumentations();

That's it. OTel auto-instruments your HTTP server, database queries, and more. No invasive code changes. You own your telemetry, not the vendor.

Dashboards per service — rate, errors and duration on one screen

Once you have metrics flowing, you need a place to look at them. Most teams use Grafana for dashboards. The best practice: one dashboard per service (or per team, depending on scale).

A solid service dashboard includes:

• Request rate – graph of requests per second over time
• Error rate – percentage of requests with status code 4xx or 5xx
• Latency – p50, p95, p99 response times
• Resource usage – CPU, memory, disk per instance
• Custom metrics – whatever matters to your business (order count, payment processing time, etc.)

Each panel should have threshold lines and alert rules attached. When error rate crosses 5%, you want to know. When p99 latency jumps from 100ms to 500ms, you want to know.

Below is a Grafana query you might use to graph error rate:

// Prometheus PromQL
rate(http_requests_total{status_code=~"5.."}[5m]) 
  / 
rate(http_requests_total[5m]) 
  * 100

This calculates: (errors in the last 5 minutes) / (total requests) × 100 = error percentage. You can alert when this exceeds 1%.

The dashboard should be simple enough that you understand what's happening in 10 seconds. If you're adding too many panels, you've lost the plot. Keep it focused on RED metrics plus a handful of domain-specific metrics. For investigation, jump into Jaeger traces or Loki logs.

Common mistakes

• Logging everything. "We log at debug level in production for better visibility." No. This tanks performance and fills your storage. Log strategically; use traces for detail.
• No trace IDs in logs. You'll spend hours correlating events across services because you didn't include trace IDs in your structured logs. It's a five-minute fix now; prevent the pain later.
• Metrics without context. Metrics without labels (service name, route, status code) are nearly useless. "error_count went up" tells you nothing. "error_count for the /payment endpoint increased" tells you everything.
• Sampling traces uniformly. Sampling 1 in 100 requests means 99 errors go untraced. Use head-based sampling for high-volume services, tail-based sampling for errors. Or use both.
• Ignoring cold starts and dependencies. Your dashboard looks great, but you only instrumented your code. What about the database response time? The message queue? Instrument end-to-end, not just your service.
• Not testing alerts. An alert that fires but no one knows how to respond to it is worse than no alert. Test them. Document runbooks. Make sure the alert message tells you what to do.

Key takeaways

• Metrics answer "is something wrong?", traces answer "where is it slow?", logs answer "why did it happen?" Use all three.
• Correlate logs, metrics, and traces via trace IDs. Every log line and metric should carry the trace ID.
• Use OpenTelemetry for instrumentation. It's vendor-neutral, industry-standard, and future-proof.
• Build one dashboard per service showing RED metrics (rate, errors, duration). Keep it simple; use traces and logs for deep dives.
• Log strategically (errors, warnings, business events). Use structured logging (JSON) so logs are searchable. Don't log everything; that's what traces are for.
• Sample traces adaptively: high-volume traffic at 1%, errors at 100%, database slow queries at 100%.
• Store traces for at least 24–48 hours, logs for at least 7 days, metrics indefinitely (or for 1–2 years at reduced granularity).