Writing runbooks: Monitoring best practices explained

Writing runbooks: Monitoring best practices explained

You've just received an alert at 3 AM that your database replication lag has exceeded 30 seconds. Your team scrambles—someone checks logs, another restarts services, a third digs through dashboards. Two hours later, you've fixed the issue, but you've wasted time on steps someone already knew. Writing runbooks is the practice of documenting exactly how to respond to alerts before they fire, so the next person (or you, exhausted at 3 AM) can fix problems faster. A runbook transforms an alert from a confusing signal into a structured path to resolution.

Throughout this series on Prometheus, Grafana, and monitoring best practices, we've covered how to instrument code, design dashboards, and configure alerts. Today, we're focusing on the critical bridge between detection and resolution: the runbook. Without runbooks, your monitoring system becomes a noise generator. With them, it becomes a teaching tool that accelerates your team's response time and reduces cognitive load during incidents.

What the alert means

An alert firing is a symptom, not a diagnosis. Before anyone can fix anything, they need to understand what the alert is actually telling them. The first section of your runbook must decode the alert's technical meaning into human terms that both the on-caller and domain experts can understand.

Start with the alert name and threshold. If your alert is HighDatabaseReplicationLag, explain that this fires when replication lag exceeds 30 seconds—but go further. Explain what replication lag is if your team includes newer engineers. Explain why 30 seconds matters for your business: if you have strong consistency requirements, 30 seconds might mean users can't immediately see their own data after writes. If you run eventual-consistency, it might barely matter.

Here's a concrete runbook opening for database replication lag:

alert_name: HighDatabaseReplicationLag
threshold: 30 seconds
what_it_means: |
  The primary database is not keeping replicas synchronized.
  When replication lag exceeds 30 seconds, read-after-write
  queries may return stale data. This affects order status,
  inventory counts, and user profile reads.
business_impact: |
  CRITICAL: Orders placed in the last 30 seconds might show
  as pending longer than expected. Inventory reconciliation
  queries might return outdated counts.
technical_meaning: |
  The Write-Ahead Log (WAL) on the primary is not being
  applied quickly enough on replicas. This indicates either
  network issues, replica CPU/IO saturation, or a backup
  job contending for resources.

Notice we're not just saying "replication lag is high." We're connecting it to real user impact and business consequences. This context helps the on-caller prioritize—is this a page-all-hands situation or a gentle nudge to investigate?

How to check impact

Once someone understands what an alert means, they need to know whether it actually matters right now. A database replication lag alert might fire during a scheduled backup, where it's expected and harmless. Or it might indicate a failing replica that's actively harming users. Your runbook must provide steps to assess the real-world impact before jumping into fixes.

This section should include specific queries, dashboard links, or log searches. Make it easy. If someone has to think hard or guess at what to check, they'll skip this step and just restart things.

For the replication lag example:

impact_check:
  step_1: |
    Open Grafana dashboard: Monitoring > Databases > Replication Health
    Look at the "Replica Lag Distribution" panel.
    If only one replica is lagging: isolated issue, lower urgency.
    If all replicas are lagging: systemic issue, higher urgency.
  
  step_2: |
    Check if a backup job is running:
    ssh prod-db-01
    ps aux | grep pg_basebackup
    If output shows a backup, this is expected. Proceed to step_3.
  
  step_3: |
    Query the primary for active connections:
    SELECT count(*) FROM pg_stat_activity WHERE state = 'active';
    Baseline should be ~50-80 connections.
    If > 200: we're CPU-bound, slow queries are piling up.
  
  step_4: |
    Check for long-running queries:
    SELECT pid, usename, query, query_start, state
    FROM pg_stat_activity
    WHERE query_start < now() - interval '5 minutes'
    ORDER BY query_start;
    These are blocking replication. Check if they're
    user-facing or batch jobs we can cancel.
  
  severity_matrix:
    one_replica_lagging: "Low - proceed with caution, do not restart"
    all_replicas_plus_backup: "Medium - wait for backup to complete"
    all_replicas_high_connections: "High - investigate slow queries"
    all_replicas_normal_load: "Critical - possible network issue"

This section gives the on-caller a structured investigation path. They're not guessing; they're following a checklist that leads to a severity assessment.

Common causes and fixes

Now that the impact is clear, it's time to fix it. A runbook should list the most common causes in order of likelihood, along with a fix for each. This is where you leverage the collective knowledge of your team—the issues you've debugged before.

For replication lag, the causes might be:

common_causes:
  
  cause_1_backup_job:
    symptom: "Backup running, all replicas lagging equally"
    fix: |
      This is expected during backups. The alert will auto-resolve
      once pg_basebackup completes. Estimated time: 20-30 minutes.
      If it takes longer than 45 minutes, check storage I/O:
      iostat -x 1 5 | grep sda
      If %util is > 90%, storage is saturated. Escalate to storage team.
    time_to_resolve: "20-45 minutes, no action needed"
  
  cause_2_slow_query:
    symptom: "Primary has 200+ active connections, long-running queries"
    fix: |
      Identify the slow query:
      SELECT pid, query FROM pg_stat_activity
      WHERE query_start < now() - interval '5 minutes';
      
      If it's a batch job (backup, analytics):
      SELECT pg_terminate_backend(pid) WHERE ...;
      
      If it's user-facing (stuck transaction):
      Contact the application team immediately.
      Do not kill the query. It may leave the app in a bad state.
    time_to_resolve: "5-15 minutes with human decision"
  
  cause_3_network_issue:
    symptom: "All replicas lagging, primary load normal, no backup"
    fix: |
      Check network connectivity:
      ping -c 5 prod-db-replica-01
      mtr -c 10 prod-db-replica-01
      
      Look for packet loss (> 1%). If found:
      - Check security group rules in AWS
      - Verify no DDoS mitigation is blocking
      - Contact network team
      
      If ping is fine, check replica disk space:
      ssh prod-db-replica-01
      df -h | grep /var/lib/postgresql
      If > 85% used, contact storage team.
    time_to_resolve: "10-30 minutes with networking help"
  
  cause_4_replica_restart:
    symptom: "One replica lagging, others normal"
    fix: |
      The replica likely crashed or was restarted.
      Check replica logs:
      ssh prod-db-replica-02
      tail -100 /var/log/postgresql/postgresql.log
      
      Look for "FATAL" or "ERROR" entries.
      If you see "out of memory," the replica needs more RAM.
      Contact infrastructure team.
      
      If logs look clean, the replica is replaying WAL.
      This is normal after a restart. It will auto-resolve
      as it catches up.
    time_to_resolve: "5-30 minutes, auto-resolves"

Notice each cause includes a symptom to match against, a step-by-step fix, and an estimate of how long resolution should take. This transforms debugging from exploratory to directed.

Who to contact

Some alerts need more than the on-caller can provide. Your runbook should be clear about when to escalate and to whom. Including contact information, expected response time, and what information to have ready prevents fumbling for Slack handles at midnight.

escalation:
  
  level_1_on_caller:
    responsibility: |
      - Impact check (is this real?)
      - Backup job diagnosis
      - Basic query investigation
    should_escalate_if: |
      - Unable to identify cause within 15 minutes
      - Multiple replicas down
      - Production data loss suspected
  
  level_2_database_team:
    slack_channel: "#database-oncall"
    expected_response: "15 minutes"
    what_to_provide: |
      - Grafana dashboard link showing the alert
      - Output of:
        SELECT * FROM pg_stat_replication;
      - Recent slow query log entries
      - Whether backup is running
    pagerduty_escalation: "Use schedule: Platform > Database"
  
  level_3_infrastructure_team:
    trigger_if: |
      - Network diagnostics show packet loss
      - Disk space critically low
      - Replica hardware failure suspected
    slack_channel: "#infrastructure-oncall"
    expected_response: "30 minutes"
    what_to_provide: |
      - Network trace output from mtr
      - Disk usage output from df -h
      - replica hostname and last restart time

By being explicit about escalation paths, you reduce the on-caller's decision burden. They know exactly when to involve the database team, exactly what information to have ready, and which team owns which layer of the stack.

Common mistakes

Mistake 1: Runbooks written too late. Teams often write runbooks after an incident has already happened and caused damage. By then, context is fuzzy, and the writer is defensive. Instead, write runbooks when you deploy the alert, before anything goes wrong. You'll make better decisions.

Mistake 2: Runbooks that are too generic. A runbook that says "check logs" or "restart the service" is worse than no runbook. It wastes time. Be specific: which logs, which file path, which restart command. Include actual code or queries.

Mistake 3: Forgetting to test the runbook. The worst moment to discover your runbook is outdated is during an incident. Test your runbooks quarterly. Simulate the alert, follow the steps as written, and update anything that's stale. This also teaches newer engineers how the system actually works.

Mistake 4: No severity guidance. A runbook that doesn't help the on-caller decide "Is this a wake-up-the-VP situation?" creates unnecessary panic. Always include business impact and severity assessment steps upfront.

Mistake 5: Assume readers have context. A runbook written by the person who designed the system assumes everyone else knows what they know. Write for someone who's just joined the team and is on-call at 3 AM. Assume nothing.

Key takeaways

• A runbook transforms an alert into a structured response path. Structure it as: what the alert means, how to check impact, common causes and fixes, and who to contact.
• Decode alerts into business impact, not just technical thresholds. Explain why the threshold matters and what users experience when it's breached.
• Provide impact-check steps as specific queries, commands, or dashboard links. Help the on-caller decide severity before taking action.
• List the three most common causes first, with symptoms and fixes for each. Be concrete: include actual commands and code, not generic guidance.
• Make escalation paths explicit. Include Slack channels, expected response times, and exactly what information to provide when handing off.
• Write runbooks when you deploy alerts, not after incidents. Test them quarterly and update them when the system changes.
• Avoid generic advice, missing severity guidance, and undocumented escalation paths. These make incidents harder to resolve, not easier.