Alert fatigue: how to fix it — best practices explained

Alert fatigue: how to fix it — best practices explained

You wake up to 47 Slack messages from your monitoring system. Most of them resolve themselves within minutes. By the time you've scrolled through half of them, you're already dismissing notifications without reading. This is alert fatigue, and it's one of the most damaging problems in modern observability. When your team ignores alerts because there are too many, the one that actually matters will slip through unnoticed.

Alert fatigue doesn't just waste time—it erodes trust in your monitoring system. Engineers stop checking dashboards, on-call rotations become demoralizing, and real incidents get buried in noise. The solution isn't to turn off alerts or lower your standards. It's to be deliberate about what you measure and when you interrupt people. We'll walk through the practical strategies that actually work.

Alert on symptoms users feel

The most common mistake is alerting on metrics that don't matter to anyone but the infrastructure team. High CPU usage on a database server sounds important until you realize it can spike to 90% and still serve requests in 10 milliseconds. Meanwhile, a 100-millisecond response time degradation—something your users definitely notice—might not trigger any alert at all.

Start by thinking like a customer. What outcomes do they depend on? Page load time. Transaction success rate. API availability. The ability to complete a checkout. Those are the symptoms that matter. If your database CPU is high but users aren't affected, it's not an emergency. If users are seeing errors, it is.

Consider a payment processing system. Instead of alerting on database replication lag in milliseconds, alert when the percentage of transactions completing within SLA drops below 99.5%. Instead of alerting on queue depth, alert when payment callbacks are delayed by more than 5 minutes and users can't confirm their purchase status.

This shift requires you to instrument your applications for business outcomes, not just infrastructure metrics. Add custom instrumentation to track the metrics your users care about:

groups:
  - name: payment_experience
    rules:
      - alert: PaymentLatencySLOBreach
        expr: |
          (
            sum(rate(payment_requests_duration_seconds_bucket{le="2"}[5m])) 
            / sum(rate(payment_requests_duration_seconds_count[5m]))
          ) < 0.995
        for: 2m
        annotations:
          summary: "Less than 99.5% of payments completing within 2s SLA"
          
      - alert: PaymentErrorRateHigh
        expr: |
          sum(rate(payment_requests_total{status="error"}[5m])) 
          / sum(rate(payment_requests_total[5m])) > 0.01
        for: 1m
        annotations:
          summary: "Payment error rate exceeds 1% threshold"

Notice the alerts are tied to user-visible outcomes. The payment latency alert fires when the SLA is breached, not when latency crosses an arbitrary threshold. The error rate alert is specific enough to matter but not so low that every blip triggers a page.

Delete alerts nobody acts on

Most alert rules in production systems are there by accident. They were created for a specific incident six months ago, they fired once or twice, and then they became noise. Nobody deleted them, so they've been sitting there ever since, training your team to ignore the alert system.

Start a audit. For each alert rule, ask: If this fired right now, what would we actually do? If you don't have a clear answer, delete it. If the answer is "check what's happening and then decide," that alert is too vague—either make it more specific or remove it.

Track alert statistics over 30 days. Count how many times each alert fired. Look at the ratio of alerts to actions taken. If an alert fired 100 times and the team took action once, something is wrong. Either the threshold is too sensitive, or the alert shouldn't exist.

Create a simple tracking sheet or query your Prometheus alert manager logs:

# Query Alertmanager to see how many times each alert was triggered
# in the last 30 days and how many were resolved by manual intervention
SELECT 
  labels->>'alertname' as alert_name,
  COUNT(*) as total_firings,
  SUM(CASE WHEN status = 'firing' THEN 1 ELSE 0 END) as unresolved,
  SUM(CASE WHEN acknowledged THEN 1 ELSE 0 END) as acknowledged_count
FROM alerts_history
WHERE timestamp > now() - interval '30 days'
GROUP BY labels->>'alertname'
ORDER BY total_firings DESC;

The alerts at the top of that list—the ones firing constantly with few acknowledgments—those are your noise. Either reconfigure them or delete them. Be aggressive. You're not losing anything valuable; you're removing obstacles to noticing real problems.

Tune thresholds with history

Alert thresholds are rarely set correctly on the first try. But most teams set them once and never adjust them. The threshold that made sense when you had 100 users breaks when you scale to 10,000. The CPU threshold that was perfect for your old hardware is useless after a migration.

Use your historical data to inform thresholds. Pull the 95th and 99th percentiles of your metrics over the last three months of normal operation. Set alert thresholds above the 99th percentile. If your API latency normally runs at p99 of 150ms, don't alert until it hits 500ms or 1,000ms—that's a real departure from normal.

This approach assumes you have baseline data. You do. You've been collecting metrics for weeks or months. Use it.

A concrete example: say you're alerting on database connection pool utilization. Instead of a hard threshold like "alert when pool is 80% full," query your actual usage patterns:

# Find the 99th percentile of normal connection pool utilization
# to inform your alert threshold
histogram_quantile(0.99, 
  max_over_time(
    db_connection_pool_utilization[30d]
  )
)

If that returns 65%, then alerting at 75% is reasonable—it's a significant deviation from your normal pattern. If it returns 92%, then 75% is too low and will create false positives.

Review and adjust thresholds quarterly. Seasonal patterns matter. Traffic that's normal in November is a disaster in December. Your thresholds should reflect your actual business patterns, not arbitrary engineering guesses.

Page only for urgent problems

Not every alert needs to interrupt someone at 3 AM. Page on incidents that require immediate human attention to prevent or mitigate user impact. Everything else should be logged, aggregated, and reviewed during business hours.

Define clear severity levels. In your alert manager, route high-severity alerts to PagerDuty. Route medium and low severity alerts to Slack channels, emails, or dashboards that the team reviews during work hours.

High severity (page immediately) should be reserved for:

• Service is fully down or unavailable to most users
• Data loss is actively occurring
• Security incident is in progress
• Revenue-impacting functionality is broken

Medium severity (Slack notification, review in morning standup):

• Degradation affecting a portion of traffic
• Non-critical infrastructure failing
• Performance below SLA but not catastrophic

Low severity (recorded in logs, visible on dashboards):

• Approaching quota limits
• Deprecated API endpoints still in use
• Non-critical health checks failing intermittently

Your alert manager configuration might look like this:

route:
  receiver: 'default'
  group_by: ['alertname', 'service']
  group_wait: 10s
  group_interval: 5m
  repeat_interval: 12h
  routes:
    - match:
        severity: critical
      receiver: 'pagerduty'
      repeat_interval: 5m
      continue: true
      
    - match:
        severity: warning
      receiver: 'slack-ops'
      repeat_interval: 2h
      continue: true
      
    - match:
        severity: info
      receiver: 'slack-logs'
      repeat_interval: 24h

Be ruthless about what qualifies as "critical." If you page someone more than once per week on average, your threshold is too low. On-call rotations only work when people trust that pages mean something genuinely urgent.

Common mistakes

Setting thresholds on derived metrics without context. Alerting on "CPU above 80%" doesn't mean anything without knowing your workload profile. Alert on the business impact instead: "requests per second below baseline" or "error rate above SLA."

Alerting on infrastructure instead of behavior. Your team doesn't care that a pod is pending. They care that users can't log in. Instrument for behavior, not infrastructure.

Creating alerts without a playbook. Every alert should have a clear playbook or runbook attached. If you don't know what to do when it fires, delete it.

Never retiring old alerts. Alerts should be treated like code—they need maintenance, reviews, and cleanup. Schedule quarterly audits to remove noise.

Alerting on everything to be safe. More alerts does not equal better safety. It equals worse safety, because your team stops paying attention. Fewer, better alerts work.

Key takeaways

• Alert on symptoms users experience, not infrastructure metrics. If the system is degraded but users aren't affected, it's not urgent.
• Audit your alert rules regularly. If nobody has acted on an alert in 30 days, delete it. Alert fatigue is worse than missing data.
• Set thresholds based on historical percentiles, not guesses. Use your actual data to determine what "abnormal" looks like for your system.
• Reserve pages for genuine emergencies. Route everything else to asynchronous channels. On-call rotation survival depends on trust that a page means something critical.
• Treat alerts like code. Review them, refactor them, retire them. Your monitoring system should be as clean as your application code.