The four golden signals: real-world lessons and examples

The four golden signals: real-world lessons and examples. What to monitor for every service.

After three weeks of building dashboards and setting up alerting rules, you might feel ready to monitor anything. But what exactly should you be watching? The answer lies in a simple framework that has shaped how we think about observability for over a decade: the four golden signals. These four metrics—latency, traffic, errors, and saturation—cut through the noise and tell you whether your services are healthy, overloaded, or failing.

We learned this the hard way. Early on, we built monitors for everything: thread counts, garbage collection pauses, HTTP header sizes. Then a service degraded silently for hours because we were watching the wrong metrics. The four golden signals give you a lens that actually matters to your users and your business.

Latency — How long requests take

Latency is the time between a request arriving at your service and a response being sent back. But here's the critical detail: you must split success latency from error latency. A request that fails fast (400ms) looks very different from one that times out (30s). If you average them, you hide the real problem.

In a real e-commerce checkout service, we track latency at multiple percentiles:

• p50 (median): 120ms under normal load
• p95: 450ms; acceptable spikes happen
• p99: 2.5s; rare but we alert on this
• p99.9: 8s; customer-visible pain; we page the on-call engineer

We also separate successful requests (HTTP 200) from failures (500s, timeouts). Here's how we instrument a NestJS service:

import { Injectable } from '@nestjs/common';
import client from 'prom-client';

@Injectable()
export class MetricsMiddleware {
  private requestDuration = new client.Histogram({
    name: 'http_request_duration_seconds',
    help: 'HTTP request latency in seconds',
    labelNames: ['method', 'route', 'status_code', 'status_class'],
    buckets: [0.01, 0.05, 0.1, 0.5, 1, 2.5, 5],
  });

  recordRequest(method: string, route: string, statusCode: number, durationMs: number) {
    const statusClass = statusCode >= 500 ? 'error' : 'success';
    this.requestDuration
      .labels(method, route, String(statusCode), statusClass)
      .observe(durationMs / 1000);
  }
}

In Prometheus, you can then query successful vs. failed latency separately:

histogram_quantile(0.95,
  rate(http_request_duration_seconds_bucket{status_class="success"}[5m])
)

The golden rule: if you only track one thing, track p99 latency for successful requests. That's what your most demanding customers experience.

Traffic — How much demand the system handles

Traffic is the volume of requests hitting your service. It's your first early warning that something unusual is happening—either a spike in legitimate demand or an attack.

In our ride-hailing platform, we track requests per second (RPS) by endpoint:

• Booking endpoint: 2,500 RPS baseline; 5,000 during peak hours
• Driver availability check: 8,000 RPS steady
• Location update: 15,000 RPS from mobile clients

A sudden 3x spike in booking requests might mean a marketing campaign worked—or a competitor went down and their traffic redirected. Either way, you need to know before your infrastructure melts.

Track traffic with a simple counter:

private httpRequestsTotal = new client.Counter({
  name: 'http_requests_total',
  help: 'Total HTTP requests',
  labelNames: ['method', 'route', 'status_code'],
});

this.httpRequestsTotal
  .labels(method, route, String(statusCode))
  .inc();

In Grafana, graph this as rate:

rate(http_requests_total[1m])

Alert when traffic deviates from normal patterns by more than 20-30%. In SaaS, a sudden drop in API calls often signals a client integration broke; a sudden spike can mean runaway loops or DDoS.

Errors — Rate of failed requests

Errors are requests that did not complete successfully. This includes HTTP 5xx responses, timeouts, and any exception your service catches and logs. The error rate is a percentage; the absolute count of errors matters far less than the rate of change.

We distinguish error types. In a CRM service:

• 4xx errors: Client mistakes (bad input, auth failure). Usually fine; don't page on-call.
• 5xx errors: Server faults. Alert immediately if rate > 1%.
• Timeouts: Dependency failures (database, cache, external API). Page on-call if sustained.
• Logic errors: Business rule violations. May indicate a data corruption or bug.

Instrument errors at the application level:

private httpErrorsTotal = new client.Counter({
  name: 'http_errors_total',
  help: 'Total HTTP errors',
  labelNames: ['method', 'route', 'error_code', 'error_type'],
});

try {
  // handler code
} catch (err) {
  this.httpErrorsTotal
    .labels(method, route, err.statusCode, err.code)
    .inc();
  throw err;
}

Calculate error rate as a percentage:

rate(http_errors_total[5m]) / rate(http_requests_total[5m]) * 100

In production, anything above 0.1% error rate for mission-critical endpoints warrants investigation. For background jobs or analytics endpoints, 1-2% might be acceptable.

Saturation — How full the system is

Saturation measures how much of your capacity you're using. It's the hardest signal to define because "capacity" depends on your infrastructure. Common saturation metrics:

• CPU: 70%+ on a single core is a warning; 90%+ is critical.
• Memory: 85%+ means garbage collection pauses will spike; 95%+ risks OOM kills.
• Disk I/O: Queue depth > 10 means database queries are piling up waiting for disk.
• Connection pools: Database connections in use / max connections. Above 80%, you'll start rejecting requests.
• Message queues: Unprocessed messages in Kafka or RabbitMQ. Growing queue = consumer falling behind.

In our trading platform, saturation is king. We monitor:

Capture saturation from the host level:

node_cpu_seconds_total
node_memory_MemAvailable_bytes

And from the application level:

private connectionPoolGauge = new client.Gauge({
  name: 'db_connections_in_use',
  help: 'Active database connections',
});

setInterval(() => {
  const used = pool.activeConnections();
  this.connectionPoolGauge.set(used);
}, 5000);

The key insight: saturation predicts failures. Before your service crashes, it saturates. Before it saturates, latency spikes. If you catch saturation early—70% CPU, 80% memory—you have time to scale or shed load gracefully.

Common mistakes

Mistake 1: Forgetting to split latency by success vs. error. A service that fails fast hides behind averaged latency. You must split the histogram buckets.

Mistake 2: Alerting on absolute error count instead of error rate. A service handling 1M requests/sec with 100 errors/sec (0.01% rate) is fine. One handling 100 requests/sec with 100 errors/sec is on fire. Rate matters; absolute count doesn't.

Mistake 3: Ignoring saturation until it causes an outage. Saturation is a lagging indicator—by the time you notice it, failures have often already begun. Monitor early and scale preemptively.

Mistake 4: Setting thresholds without historical context. "Alert if CPU > 80%" works in development but not production. Baseline your service under normal load first. Then set thresholds at 70-80% of your observed maximum healthy state.

Mistake 5: Treating all errors equally. A 401 (unauthorized) is not the same as a 503 (service unavailable). Segment errors by type and only page on-call for the ones that matter.

Key takeaways

• The four golden signals—latency, traffic, errors, and saturation—provide a universal framework for monitoring any service.
• Always split latency by success vs. error. P99 latency for successful requests is the most actionable single metric.
• Traffic baselines let you spot anomalies early; track RPS by endpoint and alert on sustained deviations.
• Error rates (not counts) tell the real story. Segment by error type; only page on-call for 5xx and timeout errors.
• Saturation is your early warning system. Monitor CPU, memory, connection pools, and queue depth to catch problems before they cascade into outages.
• Build dashboards showing all four signals side by side. When latency spikes, check traffic (Is there a load spike?) and saturation (Are we out of resources?) simultaneously.