Burn rate alerts: Monitoring best practices explained

Burn rate alerts: Monitoring best practices explained

When your SLO is burning through its error budget too fast, you need to know immediately—but not every spike warrants a page. Burn rate alerts are the bridge between noisy alerting and missed incidents, letting your team focus on real problems. In this article, we'll walk through how burn rate works, why multi-window approaches reduce false alarms, and how tools like Sloth and Pyrra automate the heavy lifting.

What is burn rate and why it matters

Your error budget is a finite resource. If you commit to 99.9% availability over 30 days, you have roughly 43 minutes of downtime to spend. Burn rate answers a simple question: at the current error rate, how long until that budget runs dry?

A burn rate of 1.0 means you're consuming your budget at exactly the rate you promised. A burn rate of 10.0 means you're burning it ten times faster than planned. If you hit a burn rate of 10 for an hour, you've torched a week's worth of budget in 60 minutes.

Without burn rate alerts, you're either drowning in alerts on every tiny error spike, or you're waiting for your SLO dashboard to turn red before you notice a problem. Burn rate gives you the Goldilocks zone: alert when errors threaten the budget, but ignore transient spikes that don't.

The math behind burn rate

Burn rate is calculated as:

burn_rate = (actual_error_rate / allowed_error_rate) over a time window

If your SLO allows 0.1% errors and you're currently seeing 1% errors, your burn rate is 10. That's a serious problem. If you're seeing 0.05% errors, your burn rate is 0.5—you're actually spending your budget slower than promised, which is ideal.

The multi-window approach: Fast and slow burn

Here's where burn rate alerting gets smart. A single threshold is dangerous: a one-minute traffic spike might hit a burn rate of 50, but if it's only for one minute, your 30-day budget barely notices. Conversely, a sustained burn rate of 2.0 over six hours is catastrophic, but might not trigger an alert watching only five-minute windows.

The multi-window strategy uses two complementary rules:

• Fast burn rule: High burn rate over a short window (e.g., burn rate > 10 for 1 hour)
• Slow burn rule: Lower burn rate over a longer window (e.g., burn rate > 3 for 6 hours)

Both conditions must be true to trigger an alert. This filters out noise: a traffic spike that burns budget fast recovers within an hour. A service degradation that bleeds budget slowly for six hours is a real, actionable problem worth waking someone up for.

Concrete example

Imagine a 99.9% SLO over 30 days (43 minutes of error budget). Your fast rule triggers on burn rate > 10 for 1 hour; your slow rule triggers on burn rate > 3 for 6 hours.

• Scenario 1 (don't alert): A deployment spike causes 2% errors for 5 minutes. Burn rate spikes to 20, but after 5 minutes it drops to 0.1%. The fast burn rule sees only 5 minutes at high burn rate, below the 1-hour threshold. No alert. The slow burn average stays low. Correct behavior.
• Scenario 2 (alert immediately): A database outage causes 5% errors. Burn rate is 50. After 30 minutes it's still at 45, then continues for another 45 minutes before recovery. Fast burn rule triggers after 1 hour at burn rate > 10. Page the team now.
• Scenario 3 (alert after 6 hours): A memory leak causes errors to rise from 0.1% to 0.4% gradually. Burn rate is 4. It stays there for 8 hours. Fast burn rule never triggers. But slow burn rule triggers after 6 hours: you've consumed your entire 30-day budget in a single day. Alert, escalate, and start incident response.

This is what multi-window alerting buys you: the ability to distinguish between problems you can ignore and problems that need your attention.

Reducing false alarms with smart thresholds

Most teams' first alerting mistake is too many alerts. Their second mistake is not tuning burn rate windows and thresholds properly. Burn rate alerting, done right, cuts false alarms dramatically.

The key insight: short spikes in error rate don't matter if your long-term budget is intact. A 99.9% SLO gives you breathing room. Use it. If your budget allows 43 minutes of downtime in 30 days, you can tolerate an outage or two. What you can't tolerate is the death of a thousand cuts—a service slowly degrading until it violates SLO.

Most false alarms come from:

• Watching only short windows: A 5-minute alert window is too twitchy. Use 1 hour minimum for fast burn.
• Thresholds that are too sensitive: If your SLO is 99%, a burn rate of 2 over 1 hour consumes only 2% of your 30-day budget. That's fine; you have 30 days to fix it.
• Single-window rules: Fast burn rules without slow burn context, or vice versa, create noise.
• Not excluding planned maintenance: If you're doing a maintenance window, disable or tune down burn rate alerts for that period.

The multi-window approach handles the first two problems. For the third, make sure your alert rule is conjunction (AND), not disjunction (OR). For the fourth, use Prometheus maintenance mode or conditional alert rules that check for maintenance labels.

Tools that do the heavy lifting: Sloth and Pyrra

Writing burn rate alert rules by hand is tedious and error-prone. Two tools have emerged as community standards: Sloth and Pyrra. Both generate Prometheus rules automatically from SLO definitions.

Sloth

Sloth is a Kubernetes-native tool that generates SLO recording rules and burn rate alerts from YAML. You define your SLO once, Sloth generates the Prometheus rules.

apiVersion: sloth.slok.dev/v1
kind: PrometheusServiceLevel
metadata:
  name: api-availability
  namespace: monitoring
spec:
  service: api
  labels:
    team: platform
  slos:
    - name: availability
      description: API request success rate
      objective: 99.9
      indicatorType: latency
      window: 30d
      indicator:
        prometheus:
          errorRateQuery: |
            sum(rate(http_requests_total{job="api", status=~"5.."}[5m]))
            /
            sum(rate(http_requests_total{job="api"}[5m]))
      alerts:
        - name: HighErrorRate
          severity: critical
          for: 1h

Sloth generates two sets of rules automatically:

1. Recording rules: Pre-computed SLO timeseries (e.g., slo:api_availability:ratio), reducing query complexity.
2. Alert rules: Multi-window burn rate alerts with sensible defaults (e.g., burn rate > 10 for 1h, burn rate > 3 for 6h).

Sloth handles the math, so you don't have to.

Pyrra

Pyrra is a more general-purpose SLO tool with a web UI and YAML-based definitions. It generates the same burn rate rules but includes a dashboard for visualizing SLO health and burn rate over time.

apiVersion: pyrra.dev/v1alpha1
kind: ServiceLevelIndicator
metadata:
  name: api-latency
spec:
  service: api
  slo: 99.0
  description: API requests under 200ms
  grouping:
    - job
  indicator:
    latency:
      threshold: 200ms
      query: http_request_duration_seconds_bucket

Pyrra's strength is the UI. You can see your burn rate, remaining budget, and time to SLO violation in real time. Both tools use the same alerting logic under the hood.

Choosing between them

If you're running Kubernetes and want a GitOps-friendly workflow, Sloth fits naturally. If you want a polished UI and dashboard out of the box, Pyrra is more batteries-included. Both are production-tested and open source. Pick the one that fits your workflow.

Common mistakes

• Using only one window: A 5-minute burn rate alert triggers on every glitch. A 30-day window misses real problems. Always use two windows.
• Setting burn rate thresholds too low: If your 30-day budget is 43 minutes and a burn rate of 1.5 for 6 hours only consumes 9 minutes, that's fine. Don't page the team. Set realistic thresholds based on your tolerance for SLO violation.
• Forgetting to exclude known noise: If every deployment causes a brief error spike, either fix the deployment or add a label to suppress burn rate alerts during your deployment window.
• Not reviewing alert performance: After a month of burn rate alerts, check how many were useful. If you're paging for things you don't care about, tune the thresholds.
• Hardcoding alert rules: Use Sloth or Pyrra to generate rules instead of writing them by hand. This keeps your SLO definition and alerting synchronized.

Key takeaways

• Burn rate measures how fast you're consuming your error budget; a burn rate of 10 means you're spending budget ten times faster than planned.
• Multi-window alerts (fast burn + slow burn) distinguish between transient spikes and real problems, cutting false alarms dramatically.
• Fast burn rules (high burn rate over 1 hour) catch sudden outages; slow burn rules (moderate burn rate over 6 hours) catch creeping degradation.
• Tools like Sloth and Pyrra generate multi-window alert rules automatically, removing guesswork and keeping SLO definitions DRY.
• Tune your burn rate thresholds based on your SLO and tolerance for violation; what matters is that you're respecting your error budget over the SLO window.
• Review alert performance monthly: if you're paging on things you don't care about, your thresholds need adjustment.