On-call basics: what every Monitoring developer should know

Introduction

You've built solid monitoring with Prometheus and Grafana, set up alerts that actually matter, and your dashboards are telling the right story. But then the alert fires at 2 AM and suddenly none of that matters if your team doesn't know who's responsible, what to do, or how to get help. On-call basics: what every Monitoring developer should know is the difference between a sustainable rotation and burned-out engineers who dread their week on duty.

We've learned this the hard way across multiple productions systems—from e-commerce platforms that can't afford downtime during peak seasons to SaaS products where reliability is the core value proposition. Being on call shouldn't feel like Russian roulette. It should feel like a well-rehearsed process where responsibilities are clear, runbooks guide your actions, and escalation paths keep you from drowning alone.

Clear rotation: Primary and secondary with handoffs

The foundation of sustainable on-call is knowing exactly who's responsible at any given moment. A clear rotation eliminates confusion, prevents both the "someone else will handle it" paralysis and the "everyone panics" chaos.

We typically use a two-tier system: a primary on-call engineer who owns the initial response, and a secondary who jumps in after a defined period or if the primary is already dealing with another incident. This isn't bureaucracy—it's survival. When you're deep in a production fire, you can't also be fielding your first page.

Here's what a practical weekly rotation looks like:

Keep rotations short and predictable. A full week works for most teams—long enough to handle the systems you own, short enough that you're not exhausted before handoff. Always run explicit handoffs: a Slack message, a brief call, or a document that says "here's what's been quiet, here's what's been flaky, here's what I'm watching." This five-minute conversation prevents the new primary from walking into blind spots.

Update your alerts and on-call contact info automatically. Use PagerDuty, Opsgenie, or Incident.io—whatever your stack is—but make sure your alert routing always knows who's on call right now. Manual updates are a bug waiting to happen.

Runbooks: Every page links to steps

An alert without a runbook is a fire alarm with no map to the exit. The moment your phone buzzes, you're already stressed. Your runbook is your co-pilot for the next thirty minutes.

Every alert that pages you should have a linked runbook. That runbook should answer:

• What does this alert mean? Not "HighErrorRate" but "more than 5% of requests to the payment API are timing out."
• Why might this be happening? List the most common causes: database slow queries, external API degradation, resource exhaustion, deployment issues.
• What do I check first? Give the exact queries, commands, or dashboard links they should look at in order.
• What are the quick fixes? Restart the service, clear the cache, scale up, rollback the last deployment—whatever is safe and fast.
• When do I escalate? If X or Y is true, don't keep troubleshooting alone.

Here's a concrete runbook structure we've found works in practice:

# Runbook: High Error Rate in Payment Service

alert: PaymentServiceHighErrorRate
severity: Critical
threshold: Error rate > 5% for 5 minutes

## What this means
The payment API is returning more errors than expected. Customers may fail to complete purchases.

## Common causes
1. Database connection pool exhausted
2. External payment processor (Stripe/Square) is slow or down
3. Recent deployment introduced a bug
4. Memory leak in the payment worker

## Immediate checks
1. Open dashboard: https://grafana.internal/d/payments-overview
2. Check payment worker logs: kubectl logs -l app=payment-worker --tail=100
3. Query database: SELECT count(*) FROM pg_stat_activity WHERE state = 'active'
4. Check Stripe status: https://status.stripe.com

## Quick fixes (in order of safety)
1. Check if a deployment happened in the last 5 minutes. If yes, rollback via ArgoCD.
2. If error rate is still high, restart payment-worker: kubectl rollout restart deployment/payment-worker
3. If database connections are high, scale up the worker: kubectl scale deployment/payment-worker --replicas=3
4. If external API is down (check status page), set feature flag to queue payments: kubectl set env deployment/payment-worker QUEUE_PAYMENT_REQUESTS=true

## Escalation
- If error rate persists after restart, page the database team (see escalation section)
- If Stripe is down, customer impact is limited; document and follow up
- If you've been troubleshooting for 15+ minutes, page the backend lead

Store runbooks in version control or your wiki, not in someone's head. Link to them from your alerting tool. Many teams embed the runbook URL directly in the alert message so it's one click away during an incident.

Escalation: Know when and how to escalate

A good on-call engineer knows when to ask for help. Escalation isn't failure—it's following the playbook.

Define clear escalation triggers and paths. Don't wait until you're panicking. Agree beforehand: "If this thing is still broken after 15 minutes of troubleshooting, we page the database team. If we can't reach them, we page the engineering director."

Here's what an escalation matrix might look like for a mid-size team:

• Page secondary after: Primary hasn't acknowledged within 5 minutes, or primary is already dealing with a different page.
• Page database team after: Error is traced to database performance and primary + secondary can't resolve within 15 minutes.
• Page infrastructure after: Error is traced to Kubernetes, AWS, or network and primary + secondary can't resolve within 15 minutes.
• Page engineering lead after: Multiple systems are down or we've hit 30 minutes on any critical incident without resolution.
• Page the CTO after: Multiple critical systems are down for 1+ hour or customer data integrity is at risk.

Make escalation friction-free. Include the escalation contact in your alert routing. Have a Slack channel for active incidents where escalations are logged automatically. The goal is to move from "I'm stuck alone" to "I've got backup" as fast as possible.

Review: Weekly review of pages and noise

On-call is only sustainable if you learn from it. A weekly review—15 minutes, same time every Monday—turns incidents into data.

Pull up the on-call summary: How many pages fired? Which alert fired most often? Which took longest to resolve? Which were false positives? This is where you find your biggest wins:

• That alert that fired three times this week for the same flaky integration? Tune it or add a runbook for the workaround.
• That page that took 45 minutes to resolve? The runbook was incomplete; update it with the actual steps that worked.
• That alert that woke someone up every night but turned out to be a test? Disable it or move it to a metric-only dashboard.
• That escalation that happened twice? Make the runbook clearer so it doesn't happen next week.

Keep a simple log. Many teams use a Slack channel or a Google Doc:

## Week of Jan 8
- PaymentServiceHighErrorRate: 3 pages, avg resolution time 8 min (Stripe was slow)
- DatabaseConnectionPoolExhausted: 1 page, resolved by scaling workers
- FalsePositive_DiskSpace: 2 pages, alert threshold too aggressive, adjusted

Actions:
- Lower DiskSpace alert threshold by 10% to reduce noise
- Add note to PaymentServiceHighErrorRate runbook: check Stripe status first
- Schedule follow-up with database team on connection pooling strategy

This review is non-judgmental. We're not blaming the on-call engineer; we're fixing the systems that made them work harder than necessary. After a few weeks, you'll notice patterns: certain systems are chronically flaky, certain runbooks are outdated, certain thresholds are wrong. Fix those and your on-call rotation gets better every week.

Common mistakes

• No clear handoff process. The new primary doesn't know what the old primary was watching. Context is lost. Use a five-minute handoff meeting or document.
• Runbooks that are out of sync with reality. If the command in the runbook no longer works, nobody on the team trusts it. Keep runbooks reviewed and updated every quarter.
• Escalation paths that are undefined. When the primary is stuck, nobody knows who to call. Define thresholds and contacts upfront and publish them.
• Skipping the weekly review. Without feedback, you learn nothing. The same noisy alerts will page you for years.
• Using on-call as punishment. If teams dread their week, they'll miss pages or ignore alerts. Rotate fairly and make on-call sustainable by keeping pages rare and runbooks good.
• Alerts paging for events that don't need immediate action. If your secondary is always called because your thresholds are too sensitive, you've created noise that makes people stop caring when a real problem happens.

Key takeaways

• A clear rotation with explicit handoffs keeps responsibility obvious and context from getting lost.
• Every page must link to a runbook that answers what, why, how, and when to escalate.
• Escalation paths should be defined and friction-free; getting help is faster than staying stuck.
• Weekly reviews of pages and resolutions turn incidents into data and drive continuous improvement.
• Sustainable on-call means your monitoring team stays sharp and your system reliability stays high.