Incident investigation with metrics: a step-by-step guide

An alert fires at 3 AM. Your error rate just jumped to 12%. The team scrambles. But where do you actually start looking? Incident investigation with metrics: a step-by-step guide turns that chaos into a methodical hunt. We'll walk through how to move from a raw alert to root cause using the tools already in your Prometheus and Grafana stack.

Start with the symptom — Which SLO or golden signal is affected?

The first thing we do when an alert lands is resist the urge to SSH into production. Instead, we look at which SLO or golden signal actually broke. The four golden signals—latency, traffic, errors, and saturation—give us a language for what "broken" means.

Let's say our alert says error_rate_5m > 5%. We open Grafana and immediately check:

• Is error rate really spiking, or is it a noisy alert threshold?
• Which HTTP status codes are we seeing? 5xx only, or mixed with 4xx?
• Is latency climbing too?
• Did traffic volume change at the exact same moment?

This matters because it shapes your next moves. A spike in 429 (rate-limit) errors means you were throttled. A spike in 503 means a dependency went down or your service is overwhelmed. A spike in 500 means your code is crashing.

We pull up a dashboard with these four signals displayed over the last 30 minutes. If latency spiked 2 minutes before error rate, that's a clue—you're likely hitting a timeout threshold. If they spiked together, it could be a deploy, a traffic surge, or both.

Narrow down — Break down by route, instance and version

A global error-rate spike is a starting point, not an answer. We now drill down into the dimensions that matter: which routes are affected, which instances, and if you've deployed recently, which versions.

In Prometheus, we use label-based breakdowns. Your metrics should already have labels like route, instance, version, and environment. A typical query looks like:

rate(http_requests_total{status=~"5.."}[5m]) by (route, instance, version)

Run this and you immediately see whether all routes are failing or just one. One instance or all of them? All versions or just the newest?

Real scenario: we had a spike where /api/checkout was hitting 18% errors, but /api/products was clean at 0.2%. That told us the issue was in checkout logic, not a database timeout affecting the whole service. We'd already saved 10 minutes by not chasing database metrics.

If it's isolated to a single instance, you're looking at that instance's logs or configuration. If it's all instances running the new version but the old version is fine, your deploy introduced a regression. If all instances and all versions are affected, it's external—a dependency, network issue, or shared resource.

At this stage, open a second Grafana window showing the same breakdown for your direct dependencies: database, cache, message queue, third-party API. Did their error rate or latency spike at the same time?

Correlate — Deploys, traffic spikes and dependency errors

Now we layer in context. Did something change? Three things to check immediately:

1. Deploys: Did a deploy go out in the last hour? Check your deployment system's timeline or add deploy markers to Grafana. Many teams use annotations or a separate deploy-events metric scraped from their CI/CD tool.
2. Traffic spikes: Did legitimate traffic surge? Black Friday, a viral post, a customer's batch job? Check the request_total metric. A 5x traffic spike hitting unscaled infrastructure explains everything.
3. Dependency errors: Did a downstream service degrade? Check error rates, latencies, and connection pool exhaustion for every system you call.

In Grafana, we create a dashboard showing our four golden signals alongside deploy timestamps and dependency health. The moment you see error rate spike and a deploy marker at the same time, the investigation narrows dramatically.

Here's a practical example: we saw error_rate jump from 1% to 8% at 14:32 UTC. A deploy went out at 14:31 UTC. Immediately, we checked that version's code changes, metrics by version, and logs from that instance pool. Within 2 minutes we found a missing null check introduced in that deploy. Reverted it, error rate dropped back to 1.2% in 30 seconds. The whole investigation took under 5 minutes because we had the timeline context.

If there's no deploy and traffic is stable, look harder at dependencies. Run a query like:

rate(upstream_errors_total[5m]) by (upstream_service)

If your database had a connection pool exhaust event or your Redis cluster had a failover, that's your smoking gun. The cascade may have started there.

Jump to traces and logs — Use exemplars and links for details

Metrics tell you what broke and when. Traces and logs tell you why. At this point, we jump into distributed tracing using the exemplar feature.

Prometheus exemplars link a metric spike to a real trace ID. In Grafana, when you hover over a spike in your error-rate graph, you can click through to a trace. That trace shows you the exact request path, which service called which, where the latency lived, and where the error happened.

Configure your instrumentation to record exemplars:

http_requests_total
  .labels(route="/api/checkout", status="500", trace_id=span.trace_id)
  .inc()

When you click the exemplar in Grafana, you land in your trace backend (Jaeger, DataDog, etc.) and see the full waterfall. You see that your service called the payments API, which timed out after 25 seconds, causing your checkout to fail.

From there, jump to logs. Filter by trace_id or request_id and get the raw error message, stack trace, and context. Logs answer the "why did the payment API timeout" question: it was trying to connect to a third-party service that had a DNS issue.

The chain is: metric spike → exemplar → trace → logs → root cause. This flow takes 2-3 minutes when your system is properly instrumented.

Common mistakes

Skipping the golden signals breakdown: Teams jump straight to logs without checking which signal broke. You end up reading 50,000 log lines looking for the one error that matters. Always start with metrics.

Not instrumenting by the right dimensions: If your metrics don't break down by route, instance, and version, you're flying blind. Add these labels to your metrics before you have an incident.

No correlation layer: If deploy timestamps and dependency metrics aren't visible on the same dashboard, you'll miss the connection. Build that dashboard during peace time, not during an incident.

Exemplars without links: Exemplars that don't link to traces are frustrating. Make sure your Grafana datasources are configured to talk to your trace backend, and your instrumentation actually records trace IDs in exemplars.

Treating all spikes as equal: A 1-minute blip is different from a 30-minute sustained outage. Check the duration. A 1-minute spike might be a transient hiccup you only investigate if it repeats. A 30-minute spike demands a postmortem.

Key takeaways

• Start with the golden signals (latency, traffic, errors, saturation) to understand which SLO broke, not which system.
• Drill into dimensions immediately: break down by route, instance, and version to narrow the blast radius.
• Correlate your metrics timeline with deploys, traffic patterns, and upstream errors—context wins investigations.
• Use Prometheus exemplars to link metric spikes to specific traces, then jump to logs for the full story.
• Instrument for investigation during peace time: add labels, build dashboards, and test your trace links before you need them at 3 AM.