Prometheus alerting rules: practical tips and trade-offs

Introduction

Three a.m. on a Tuesday. Your phone buzzes with an alert—the third one this week for the same non-issue. By contrast, the outage that actually mattered last month went undetected for fifteen minutes. This scenario plays out in teams across the industry, and it points to a fundamental truth: Prometheus alerting rules: practical tips and trade-offs is not just about syntax. It's about designing alerts that your team will trust enough to wake them up.

In this post, we'll walk through how to define rules that discriminate signal from noise, route alerts to the right people, and include the information humans need to respond. We'll also explore the trade-offs you'll face when tuning thresholds, durations, and labels.

Anatomy of an alert rule: expression, duration, and labels

Every Prometheus alerting rule has three core components: a PromQL expression that defines the condition, a for duration that prevents flapping, and labels that route and contextualize the alert. Let's start with a concrete example.

groups:
- name: api
  interval: 30s
  rules:
  - alert: HighErrorRate
    expr: job:http_errors:ratio5m > 0.05
    for: 10m
    labels:
      severity: page
      team: platform
    annotations:
      summary: "Error rate above 5% for 10 minutes"
      description: "{{ $labels.job }} has error ratio {{ $value | humanizePercentage }}"
      runbook_url: "https://wiki.example.com/runbooks/high-error-rate"

The expr is a PromQL query that returns a time series. In this case, we're checking a pre-computed recording rule job:http_errors:ratio5m that divides 5-minute error count by total requests. The query returns non-empty results only when the condition is true.

The for duration—here 10 minutes—is how long the expression must return a non-empty result before an alert fires. During that period, the rule is in the PENDING state. Only once the full duration has elapsed does it transition to FIRING.

Labels are key-value pairs attached to every alert instance. severity controls routing: page means wake someone up now; ticket means create a Jira ticket. The team label might feed into escalation policies. Custom labels help you slice and dice alerts in Grafana dashboards or pass context to incident management systems.

Annotations are free-form fields for human consumption. The summary appears in Slack, PagerDuty, or your alert interface. The runbook_url is critical—we'll come back to that.

The for duration: timing your detections

Setting the right for duration is a trade-off between speed and stability. Set it too low, and you'll page on every blip—a database query that takes 200ms longer than usual, a brief GC pause, a transient network hiccup. Set it too high, and you'll miss real problems.

A practical heuristic: set your duration to match your tolerance for degradation plus your detection window. If you can tolerate 5 percent error rate for 5 minutes but not 10 minutes, then set for: 5m. If you're OK with brief spikes but want to catch sustained issues, use 10 minutes.

Consider a real-world example: a payment processing service. A single failed transaction might spike your error rate to 10 percent, but you can retry. A stuck payment thread causing 50 percent errors sustained over 5 minutes is a page. You might define:

- alert: PaymentProcessingHigh
  expr: rate(payments_errors_total[5m]) / rate(payments_total[5m]) > 0.5
  for: 5m
  labels:
    severity: page
    team: payments

The rule waits 5 minutes of sustained 50 percent error rate before paging. A one-time spike to 80 percent lasting 2 minutes will not trigger—good, because you want the retry queue to handle that. But 5 minutes of steady problems gets your attention immediately.

A different story for disk usage. Disks fill slowly, and by the time they're at 90 percent, you've probably had a few hours to notice. A 1-hour for duration here makes sense—it eliminates flapping on brief write bursts:

- alert: DiskAlmostFull
  expr: node_filesystem_avail_bytes / node_filesystem_size_bytes < 0.1
  for: 1h
  labels:
    severity: ticket

Conversely, a 10-second latency spike on API calls is probably harmless noise. A sustained 500ms latency increase over 2 minutes might warrant investigation if it's affecting customers. Adjust accordingly:

- alert: APILatencyHigh
  expr: histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m])) > 0.5
  for: 2m
  labels:
    severity: ticket

Severity labels: page versus ticket routing

Not all alerts should wake people. Your routing strategy should reflect your organization's on-call structure and values.

Page (severity: page) means: this requires immediate human action and will impact customers or systems within minutes if untouched. The database connection pool is exhausted. The main API is returning 500s. A critical background job hasn't completed in hours. These wake on-call engineers at any time.

Ticket (severity: ticket) means: this is worth investigating, but it can wait until business hours or the next shift. Error rates are elevated but not critical. Memory usage is climbing. A non-critical service is down. These create Jira tickets or Slack notifications in a team channel.

A third category, warning, sometimes exists for trend-based alerts: error rates creeping up, latency drifting higher, or custom business metrics showing early warning signs. These might only create tickets or show up in a daily digest.

The key: every page should require action. If your team routinely silences or dismisses "page"-severity alerts, you've tuned your thresholds wrong. You're training the team to ignore real signals.

In a real e-commerce platform, you might define:

Runbook links: making alerts actionable

An alert without a runbook is a wake-up call with no answer to "what do I do?" At 3 a.m., your on-call engineer has seconds to decide whether to page a manager or start troubleshooting. A clear runbook cuts that friction dramatically.

Every alert should include an annotations.runbook_url that links to a wiki page or internal doc. The runbook should include:

• What this alert means: one sentence definition of the condition.
• Why it matters: what customer or system impact is likely?
• How to diagnose: what logs, metrics, or commands to check?
• How to resolve: restart service, scale horizontally, drain connections, etc.
• When to escalate: if it's still happening after 10 minutes of troubleshooting, page the on-call lead.

Here's a real example structure in your alert rule:

- alert: PostgresReplicationLag
  expr: pg_replication_lag_seconds > 60
  for: 5m
  labels:
    severity: page
    team: data
  annotations:
    summary: "PostgreSQL replication lag over 60 seconds"
    description: |
      Replica {{ $labels.instance }} is {{ $value | humanize }} seconds behind primary.
      High replication lag increases failover risk.
    runbook_url: "https://wiki.example.com/runbooks/postgres-replication-lag"

The runbook at that URL might say:

PostgreSQL Replication Lag

The standby database is falling behind the primary. This can happen if the standby is under high load or if network bandwidth is constrained.

Check: SSH to the replica, run psql -c "SELECT slot_name, restart_lsn, confirmed_flush_lsn FROM pg_replication_slots;" and compare to primary LSN. Check replica CPU and disk I/O with iostat -x 1. Look at the primary's binary log shipping rate: SELECT pg_wal_lsn_diff(pg_current_wal_lsn(), '0/0');

If replica is slow: Kill long-running queries on the replica or scale up its compute. If network is saturated, check for large table scans on the primary.

If lag persists beyond 10 minutes: Page the database on-call lead.

Without this guidance, your engineer spends 10 minutes Googling or calling teammates. With it, they're triaging in under 2 minutes.

Common mistakes

Mistake 1: alerts on single metric spikes. A 99th percentile latency spike to 5 seconds doesn't mean your service is broken—it might mean one user had a slow network. Use quantiles over 5-minute windows and require sustained breaches before paging. Recording rules help here: api:http_request_duration_p99:5m smooths noise.

Mistake 2: forgetting to test thresholds. Set a threshold in production, then wait to see how often it fires. If it fires more than once a month for a non-issue, lower the threshold or raise the for duration. Use your Prometheus query builder to backtest: run your expression against historical data to see how often it would have triggered.

Mistake 3: over-alerting on dependencies. If your API depends on a third-party payment processor and it goes down, you'll get alerts from your service about timeout errors. But your service is working correctly—it's the upstream that's broken. Filter those out or use a lower severity. Catch the upstream outage directly if you can.

Mistake 4: alerts without context labels. If your alert just says "error rate high" with no job, instance, or endpoint label, your on-call engineer will query Prometheus to find which service is affected. Add labels for job, instance, endpoint, and datacenter. These should match your label schema for easy correlation with dashboards.

Mistake 5: no silence mechanism. Sometimes you're doing a deployment or maintenance. You need a way to silence specific alerts without removing the rules. Use Alertmanager's silences feature (time-bounded) or your incident management system's mute alerts button. Temporary silence during a known issue prevents alert fatigue and keeps trust in your alerting system.

Key takeaways

• Structure every alert rule with three components: a PromQL expression that defines the problem, a for duration that prevents false positives, and labels that route and contextualize the alert.
• Set for durations based on your tolerance for sustained degradation. Disk usage can tolerate 1 hour; error rates might need 5 minutes; latency might need 2 minutes.
• Use severity: page only for issues that require immediate action and will harm customers or systems if ignored. Everything else should be a ticket or warning.
• Every page-severity alert must link to a runbook that explains what the alert means, why it matters, how to diagnose, and how to fix it.
• Backtest your thresholds against historical data. If an alert fires more than a few times a month for noise, it's tuned wrong.
• Trust is earned through consistency. Alerts that are correct build confidence; those that are wrong erode it.