100 days of Prometheus, Grafana and monitoring: the recap

Over the past 100 days of Prometheus, Grafana and monitoring, we've moved through the full lifecycle of building observability into production systems—from first metrics to debugging production incidents at scale. What started as a question of "how do we know when things break?" has become a discipline that shapes how we architect, deploy and operate our services. This recap distills the lessons that matter most, the patterns we've validated in production, and the mistakes we've learned to avoid.

Measure what users feel — Golden signals and SLOs

The hardest truth about monitoring is that most metrics don't matter. We can track CPU, memory, disk I/O, database connection pools, and dozens of other signals, yet still miss that our users are experiencing 500ms latency spikes or that payments are timing out. This is why the golden signals framework exists: focus on latency, traffic, errors, and saturation—the four signals that directly correlate to what your users experience.

But measurement alone isn't enough. We need to bind those signals to promises. That's where Service Level Objectives (SLOs) come in. An SLO is a target—typically expressed as a percentage of successful requests over a time window. If you commit to 99.9% of requests completing within 200ms over a 28-day period, that's your SLO. The difference between your SLO and 100% is your "error budget," and it's the budget that drives alert strategy and incident response priority.

Consider a typical SaaS API serving customer data. Your SLO might be:

99.9% of requests complete successfully
99th percentile latency under 200ms
Monthly error budget: 43 minutes of downtime

When you burn through that budget in the first week, you stop shipping new features and focus on stability. When you're tracking to budget, you can move faster. This changes the conversation from "we need zero errors" (impossible) to "we need to respect our commitments."

In Prometheus, you'd track this with histogram metrics and record rules:

yaml

groups:
  - name: slo
    interval: 30s
    rules:
      - record: api:request:success:ratio_5m
        expr: |
          sum(rate(http_requests_total{status=~"2.."}[5m]))
          /
          sum(rate(http_requests_total[5m]))
      
      - record: api:request:latency:p99_5m
        expr: |
          histogram_quantile(0.99, 
            sum(rate(http_request_duration_seconds_bucket[5m])) by (le))

Then in Grafana, alert when you're tracking to burn through your monthly budget faster than expected—not when a single request fails.

Labels are power and danger — Control cardinality from day one

Prometheus stores every unique combination of metric name and label values as a separate time series. If you emit a metric http_requests_total with labels for endpoint, method, status, and user_id, and you have 500 endpoints, 5 methods, 10 status codes, and millions of users, you've just created 500 × 5 × 10 × millions of time series. Prometheus will run out of memory. This is cardinality explosion, and it's the most common way teams break their monitoring infrastructure.

The rule is simple: never add unbounded labels. User IDs, request IDs, order IDs, email addresses—these are all unbounded. They grow without limit. Bounded labels are safe: HTTP methods (GET, POST, etc.), status codes (200, 404, 500), regions (us-east-1, eu-west-1), or environment (prod, staging). Use bounded labels liberally. Avoid unbounded ones entirely.

Here's what cardinality explosion looks like in practice:

yaml

# GOOD: bounded labels
http_requests_total{method="GET", status="200", region="us-east-1"}
database_queries_total{operation="insert", table="orders", status="success"}

# DANGEROUS: unbounded labels
http_requests_total{method="GET", status="200", user_id=""}
database_queries_total{operation="insert", query_hash="", customer_id=""}

If you need to track per-user metrics, emit aggregates: count of requests per user, p95 latency per user, grouped into buckets. Don't emit raw user IDs in labels. If you need request-level detail, that's what structured logging and distributed tracing are for—not metrics.

Monitor your own cardinality. Most teams ignore this until Prometheus crashes at 3am:

yaml

# Query to check cardinality growth
count(count({__name__=~".+"}) by (__name__))

Run this daily. If it spikes, investigate before it becomes an incident.

Alert with intent — Symptoms, runbooks and burn rates

The worst alerts are the ones that fire at 2am and require 20 minutes of investigation to determine if anything is actually wrong. We've all been there. The fix is to alert on symptoms, not causes.

A cause-based alert: "CPU usage is above 80%." So what? CPU can spike during legitimate traffic. Maybe you're fine. Maybe you're about to crash. The alert is noise.

A symptom-based alert: "Error rate has exceeded 1% for the past 5 minutes." This is something users are experiencing right now. If your SLO is 99.9% success, then 1% errors means you're burning your error budget fast. This deserves a page.

Write alerts that answer the question: "Is my user experience degraded right now?" If the answer could be "maybe" or "probably not," keep iterating.

For each alert, write a runbook. A runbook is a decision tree: what checks do I run first? What are the common causes? What's the escalation path? Without a runbook, every alert becomes an archaeological dig.

Example runbook for high error rate:

Alert: HighErrorRate (error_rate > 1% for 5 minutes)

First checks: Is traffic elevated? Check the traffic graph. If traffic is normal and errors are high, proceed. If traffic spiked, this might be expected—check the deployment timeline.

Common causes: Recent deployment (check git log), database connection pool exhaustion (check pool metrics), external dependency timeout (check dependency latency).

Resolution: If recent deployment, rollback. If database pool is maxed, investigate query performance or connection leak. If external dependency is slow, page the owning team.

Finally, use burn rate alerts. Your SLO defines an acceptable monthly burn rate. If you're burning your monthly error budget in one day, that's a pages-level incident. Burn rate alerts fire based on how fast you're consuming your error budget, not on absolute thresholds:

yaml

groups:
  - name: slo_alerts
    rules:
      - alert: HighErrorBurnRate
        expr: |
          (1 - (
            sum(rate(http_requests_total{status=~"2.."}[5m]))
            /
            sum(rate(http_requests_total[5m]))
          )) > 0.10
        for: 5m
        annotations:
          summary: "Burning error budget at 10x the acceptable rate"

Connect the signals — Metrics, logs and traces together

Metrics give you the big picture. A line on a graph showing that error rate spiked at 14:32 UTC. But metrics don't tell you why. You need logs and traces to understand the why.

Logs answer: What happened? Structured logs with consistent fields let you search quickly. When error rate spikes, you query your logs for errors in that time window, group by error type, and find that 95% of errors are "connection timeout on customer-data-service."

Traces answer: How did this request flow through the system? They show you the path from the API gateway through your service, into the database, and back. If a request is slow, a trace shows you exactly where it spent its time: 50ms in your code, 950ms in the database query.

The best teams connect these three. In Grafana, when you alert on high latency, you should be able to click through to relevant logs and traces. When you find an error in logs, you should see the corresponding trace. When you're looking at a trace, you should see the metrics context around that time.

This requires:

A common request ID propagated through metrics, logs, and traces
Instrumentation in every layer: application code, API gateways, databases, message queues
Integration between systems: Grafana can link to Loki (logs) and Jaeger (traces) via URL templates

In practice, your Grafana dashboard for a service should have panels for:

Request rate and latency (metrics)
Error rate broken down by error type (metrics + logs)
Recent errors with links to traces (logs)
Dependency latency and health (metrics)

When something breaks, you move from metric to log to trace to fix in minutes, not hours.

Common mistakes

Alerting on everything: If you have more than five alerts per service in production, most of them are probably noise. Ruthlessly prune low-value alerts.
Ignoring cardinality until it's too late: Add cardinality monitoring from day one. Check it weekly.
SLOs without consequences: An SLO that doesn't change your team's behavior isn't an SLO. Define them, track them, act on them.
Metrics without context: A dashboard full of graphs is theater. Each panel should answer a specific question for the person reading it.
No runbooks: If you can't explain what an alert means and what to do about it in two paragraphs, the alert shouldn't exist.

Key takeaways

Golden signals and SLOs let you measure what users actually experience, not just system health.
Cardinality control is a day-one concern, not an afterthought. Bounded labels only.
Alert on symptoms and user impact. Write runbooks. Use burn rate alerts to prioritize.
Metrics, logs, and traces are useless in isolation. Connect them.
Observability is infrastructure for decision-making. Every chart should answer a question, every alert should drive action.

100 days of Prometheus, Grafana and monitoring: the recap