Grafana OnCall and incident tools: a step-by-step guide

Introduction

When alerts start firing at 3 AM, you need more than just notifications—you need a system that knows who to wake up, how to route the incident, and where to keep the record straight. Grafana OnCall and incident tools: a step-by-step guide shows you how to build that backbone into your monitoring infrastructure. In this piece, we'll walk through schedules and escalations, alert grouping, Slack integration, and incident timelines so your team can respond with confidence instead of chaos.

Why incident management matters

Most teams start with Prometheus and Grafana to detect problems. But detection is only the first mile. The real challenge is getting the right person notified at the right time, making sure critical alerts don't drown in noise, and keeping everyone on the same page once an incident starts. Without a dedicated incident tool, you end up with:

• Multiple people responding to the same alert because no one knows who owns it
• Important incidents hidden under hundreds of noisy notifications
• No audit trail of who said what and when during a crisis
• Lengthy post-mortems because no timeline exists

Grafana OnCall solves these problems by sitting between your alerts and your team.

Schedules and escalations

At the core of on-call management is knowing who is responsible right now. Grafana OnCall lets you define schedules that rotate through your team members, and escalation policies that ensure no alert goes unanswered.

Setting up a basic schedule: Start by defining your team's shift pattern. You might have a primary on-call engineer for the business day, a secondary for nights, and a manager escalation if no one responds in 15 minutes.

Schedule:
  name: "Backend Team On-Call"
  timezone: "UTC"
  layers:
    - name: "Primary"
      start_date: "2025-01-13"
      rotation_start: "09:00"
      rotation_period: "1 week"
      members:
        - alice@example.com
        - bob@example.com
        - charlie@example.com
    - name: "Secondary"
      start_date: "2025-01-13"
      rotation_start: "17:00"
      rotation_period: "1 week"
      members:
        - diana@example.com
        - eve@example.com

This setup rotates Alice, Bob, and Charlie through primary on-call duty, each taking one week. Diana and Eve cover evenings and weekends as secondary. Grafana OnCall tracks this automatically, so your alerts always know who to page.

Escalation policies keep incidents moving: Even the best on-call engineer needs backup. Define what happens if the primary doesn't acknowledge an alert within 5 minutes, or if they resolve it incorrectly. A typical escalation chain looks like:

1. Notify primary on-call (wait 5 minutes)
2. If no acknowledgment, notify secondary (wait 5 minutes)
3. If still no response, notify team lead
4. If after 15 minutes total still unaddressed, page the manager

In Grafana OnCall, you set these rules once and they apply to all incidents routed through that escalation policy. When you build new alert rules in Prometheus, you simply attach them to an escalation policy by name.

Alert grouping

Without grouping, a single issue—like a database connection pool exhaustion—can trigger hundreds of identical alerts. Your on-call engineer opens OnCall to find 200 notifications and freezes. Alert grouping solves this by intelligently merging related alerts into a single incident.

Grouping by label: Grafana OnCall groups alerts using labels you define. For example, if your Prometheus rules include labels like service, team, and region, you can tell OnCall: "Group all alerts with the same service and region into one incident."

Alert Grouping:
  grouping_field: "service"
  additional_fields:
    - "region"
    - "severity"
  group_wait_time: "30s"
  group_timeout: "5m"

With this policy, if your API service in us-east-1 triggers 50 different alerting rules at once (timeouts, memory pressure, high error rates), they all become a single incident. Your on-call engineer sees one page, not 50.

Real-world example: Your e-commerce platform's checkout service starts timing out. Prometheus fires alerts for:

• checkout_api_latency_high
• checkout_db_connections_exhausted
• checkout_api_errors_high
• checkout_payment_provider_slow

Without grouping, four separate incidents page your team. With grouping by service=checkout and region=us-west-2, all four become one incident titled "Checkout service incident in us-west-2." The on-call engineer opens it, sees all related alerts in one place, and recognizes immediately that this is a cascading failure starting with DB connections.

Integrations with Slack

Your on-call team already lives in Slack. Grafana OnCall integrates directly, so incidents appear as interactive messages right where the team works.

Setting up the Slack bot: In Grafana OnCall, go to Integrations, select Slack, and follow the OAuth flow. OnCall will create a dedicated Slack bot in your workspace. Grant it permissions to post messages, manage threads, and react to messages.

Incident notifications in Slack: When an incident fires, OnCall posts a rich notification in your designated channel (e.g., #incidents):

🚨 Critical: Database replication lag in prod-us-east-1

Service: analytics-pipeline | Severity: Critical | Assigned to: alice

[Acknowledge] [Resolve] [Escalate]

Team members click buttons directly from Slack. When Alice clicks "Acknowledge," OnCall records the timestamp and stops escalating. If she clicks "Resolve," the incident closes and the alert silences. If 15 minutes pass without acknowledgment, Slack shows a follow-up message: "No response yet. Escalating to secondary on-call."

Thread continuity: OnCall groups all updates for an incident in a single thread. As the incident progresses, team members add context, links to dashboards, runbooks, and status updates all in one place. No hunting through multiple Slack messages.

Custom message templates: You can customize what Grafana OnCall sends to Slack. Include links to dashboards, runbook URLs, or even curl commands to trigger diagnostics:

Slack Template:
  title: "Alert: {{ alert_name }}"
  text: |
    Service: {{ service }}
    Severity: {{ severity }}
    Dashboard: https://grafana.example.com/d/abc123
    Runbook: https://wiki.example.com/runbooks/{{ service }}
  color: "{{ color_by_severity }}"

Incident timelines

After an incident resolves, your team needs to understand what happened and why. Grafana OnCall maintains a detailed timeline of every action taken during the incident.

What gets recorded:

• When alerts fired and their exact values
• When the on-call engineer acknowledged
• When they added status updates or comments
• When escalations occurred
• When related alerts were grouped in
• When the incident was resolved
• Links to logs, metrics, or external references

Example timeline: Your payment service incident might look like:

Later, during post-incident review, you import this timeline into your incident analysis. You see exactly how long each phase took: detection to acknowledgment (45 seconds), acknowledgment to diagnosis (1.7 minutes), diagnosis to resolution (2.4 minutes). You identify that pod-3 had been showing memory pressure warnings for hours before the spike, which could have triggered a preemptive restart.

Exporting and reporting: Grafana OnCall can export incidents to JSON or integrate with tools like PagerDuty, Opsgenie, or your internal incident database. Many teams sync OnCall incidents to a spreadsheet or dashboard to track response times, mean time to resolution (MTTR), and escalation frequency over time.

Common mistakes

Over-grouping alerts: It's tempting to group all alerts by just service, but if your checkout service spans three regions and two deployment environments, you might get one incident combining a real production outage with a harmless staging alert. Always include environment and region in your grouping strategy.

Ignoring escalation fatigue: If your escalation policy escalates too quickly or too aggressively, your manager gets paged for every minor alert. Set realistic acknowledgment timeouts (5–10 minutes for critical alerts, 30 minutes for warnings) and test your escalation chain during a controlled drill.

Leaving Slack integration half-configured: If your OnCall bot can post messages but team members can't click buttons, incidents pile up in Slack without being acknowledged in OnCall. Test every button (Acknowledge, Resolve, Escalate) before deploying to production.

Not automating incident closure: When an alert resolves in Prometheus, tell OnCall to close the incident automatically. Otherwise, you end up with incidents that linger as "resolved" but not closed, cluttering your timeline and making metrics unreliable.

Skipping post-incident reviews: The timeline is only useful if someone reads it. Schedule brief 15–30 minute post-mortems for severity-1 and severity-2 incidents, export the OnCall timeline, and identify systemic improvements. Over time, this discipline reduces incident frequency and response time dramatically.

Key takeaways

• Schedules and escalations ensure the right person is always on-call and critical incidents don't go unanswered. Define rotations by timezone and role, and escalate to secondary or management if the primary doesn't respond within your agreed window.
• Alert grouping reduces noise by merging related alerts from the same service, environment, or region into a single incident. This keeps on-call engineers focused and prevents decision paralysis.
• Slack integration brings incidents directly to where your team works, with interactive buttons for acknowledgment and resolution. Use custom templates to include links to dashboards, runbooks, and diagnostic tools.
• Incident timelines create an audit trail for every action, supporting fast diagnosis during the incident and thorough post-incident review afterward. Export timelines to identify patterns and reduce MTTR over time.
• Test your on-call setup quarterly with scheduled drills. Verify that alerts page correctly, that escalation chains work, and that Slack integration behaves as expected.
• Treat on-call as a team responsibility, not a burden for one person. Rotate duties fairly, provide secondary coverage, and celebrate quick resolutions in your team channel.