Alertmanager configuration: a practical Monitoring guide

Alertmanager configuration: a practical Monitoring guide

By day 15, you've probably collected metrics and written alert rules—but without proper Alertmanager configuration, those alerts become noise instead of actionable signals. Routing, grouping, and silencing are the three pillars that turn raw alerts into a communication channel your team actually trusts. In this guide, we'll walk through the decisions and configurations that separate a monitoring setup that works from one that keeps you awake at 3 AM.

Routes — Send alerts to teams by labels

Routes determine where each alert goes. Think of routing as the switchboard operator of your alerts: when a database CPU spike fires, it needs to reach the database team on Slack; when a payment service is down, it needs to page the platform team immediately. Prometheus alert rules attach labels to every alert—use those labels to decide the destination.

At a basic level, your Alertmanager config has a root route with a default receiver. Child routes can override that receiver based on label matching. Here's a real example:

global:
  resolve_timeout: 5m

route:
  receiver: 'default-receiver'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 4h
  routes:
    - match:
        team: platform
      receiver: 'platform-team'
      group_wait: 5s
      repeat_interval: 2h
    - match:
        team: database
      receiver: 'database-team'
      group_wait: 15s
    - match_re:
        severity: 'critical|warning'
        service: '^(payment|checkout).*'
      receiver: 'payment-critical'
      group_wait: 0s
      repeat_interval: 1h

receivers:
  - name: 'default-receiver'
  - name: 'platform-team'
  - name: 'database-team'
  - name: 'payment-critical'

In this setup, alerts with team: platform go to the platform team receiver; alerts matching the regex for critical payment services go to a dedicated payment receiver with zero group wait—meaning they notify immediately without waiting to batch with other alerts.

The key decision here is group_by: which labels should Alertmanager use to batch related alerts together? Group by alertname, cluster, and service so that five disk space warnings on the same service appear as one notification. Avoid grouping by hostname if you have 100 instances—you'll end up with 100 separate notifications.

Grouping — Combine related alerts into one notification

Grouping is about volume control. Imagine a network partition hits and 50 application instances lose database connectivity. Without grouping, your team gets 50 individual alerts. With grouping, you get one notification: "DiskSpaceWarning on service:checkout across 50 instances."

Grouping is controlled by three timers:

• group_wait: How long Alertmanager waits before sending the first notification for a new group. Use 10–30 seconds to allow related alerts to arrive together.
• group_interval: How long between updates for an existing group. Once a group is firing, this controls how often you hear about it again.
• repeat_interval: How long until an already-notified group fires again. Set this to hours, not minutes, to avoid alert fatigue.

Here's where the math matters. Suppose you fire a CPU alert at 09:00. With repeat_interval: 4h, you'll get another notification at 13:00 if it's still firing. In a 24-hour period, you get roughly 6 notifications instead of one every minute.

In practice, we've found these settings work across most teams:

route:
  receiver: 'default'
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 10s
  group_interval: 10s
  repeat_interval: 4h

But adjust for your context. A payment team handling transaction failures might want group_wait: 0s and repeat_interval: 1h. An infrastructure team monitoring batch jobs might tolerate group_wait: 60s and repeat_interval: 24h.

Inhibition — Suppress symptoms when the root cause is alerting

Inhibition is the silent workhorse of mature monitoring. When a database server goes down, it creates a cascade: the database alert fires, connection pool alerts fire, application latency alerts fire, timeout alerts fire. Your team sees 20 notifications. Inhibition says: "If the database is down, suppress the connection pool alert—it's just a symptom."

Set up inhibition rules to match the root cause against dependent alerts. In your Alertmanager config:

inhibit_rules:
  - source_match:
      alertname: 'DatabaseDown'
    target_match:
      alertname: 'DatabaseConnectionPoolExhausted'
    equal: ['cluster', 'environment']
  
  - source_match:
      alertname: 'KubernetesNodeNotReady'
    target_match:
      severity: 'warning'
    equal: ['node', 'cluster']
  
  - source_match:
      alertname: 'ServiceUnhealthy'
    target_match:
      alertname: 'HighErrorRate'
    equal: ['service', 'environment']

Read these rules as: "When DatabaseDown fires with the same cluster as a DatabaseConnectionPoolExhausted alert, suppress the connection pool alert." The equal field specifies which label values must match between the source and target for inhibition to apply.

Inhibition is subtle. A poorly configured rule will hide real incidents. Use it for relationships you're confident about: database down implies connection failures, node not ready implies pod crashes, service unhealthy implies high error rates. Avoid broad rules like "any critical alert inhibits all warnings"—you'll miss genuine problems.

Receivers — Slack, email, PagerDuty and webhooks

Routes point to receivers; receivers define the actual delivery mechanism. Alertmanager ships with integrations for Slack, email, PagerDuty, Opsgenie, Webhooks, and more. Here's a realistic setup:

receivers:
  - name: 'default-receiver'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
        channel: '#alerts-general'
        title: 'Alert: {{ .GroupLabels.alertname }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'

  - name: 'platform-team'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
        channel: '#platform-alerts'
        send_resolved: true
    email_configs:
      - to: 'platform@company.com'
        from: 'alertmanager@company.com'
        smarthost: 'smtp.company.com:587'
        auth_username: 'alertmanager@company.com'
        auth_password: 'your-app-password'

  - name: 'payment-critical'
    pagerduty_configs:
      - service_key: 'YOUR_PAGERDUTY_KEY'
        description: 'Payment service alert: {{ .GroupLabels.alertname }}'
        details:
          severity: '{{ .GroupLabels.severity }}'
          service: '{{ .GroupLabels.service }}'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/YOUR/WEBHOOK/URL'
        channel: '#payment-critical'
        color: 'danger'

Notice the pattern: critical production issues go to PagerDuty (which escalates if no one acknowledges); platform issues go to Slack and email; general alerts go to Slack. The send_resolved: true option tells the receiver to send a message when an alert resolves—useful for teams who need to know when the incident is over.

Use templating to customize the message. {{ .GroupLabels.alertname }} gives you the alert name; {{ .Annotations.description }} gives you the description you defined in your Prometheus alert rule. This way, Slack messages are readable and actionable without requiring the team to log into Prometheus.

Common mistakes

Setting repeat_interval too low. The default is 4 hours. If you set it to 5 minutes, an alert firing continuously generates 288 notifications per day. Your team stops reading them.

Not grouping by the right labels. If you group by hostname on a 500-node cluster, you get 500 notifications. Group by service name and severity instead. Group_by should have 3–5 labels at most.

Over-inhibiting. A rule that's too broad hides real incidents. "Suppress all warnings when any critical alert fires" will mask a second, unrelated critical issue. Match inhibition rules carefully and test them.

Forgetting to test webhook integrations. Deploy Alertmanager, configure PagerDuty, and then discover the integration doesn't work at 2 AM. Send a test alert to each receiver after deployment.

Not setting group_wait correctly for the team's SLA. A support team investigating user-facing issues might need group_wait: 5s. A billing team reconciling transactions at end-of-day can tolerate group_wait: 5m. Think about urgency.

Key takeaways

• Routes match alert labels and send to receivers; use label-based routing to separate concerns by team or service.
• Group_by, group_wait, and repeat_interval control volume; tuning these prevents alert fatigue.
• Inhibition rules suppress symptom alerts when root cause alerts fire, keeping signal high.
• Receivers are channels (Slack, PagerDuty, email, webhooks); pair critical issues with escalation (PagerDuty), normal issues with team channels (Slack).
• Test all receiver integrations in a staging environment before production—misconfigured webhooks are silent failures.
• Review your routing and grouping settings after the first month; teams' needs evolve as your system grows.