rate vs irate vs increase: practical tips and trade-offs

Introduction

When you're building production monitoring for a distributed system, choosing the right aggregation function can mean the difference between actionable alerts and noise. In this post, we'll cover rate vs irate vs increase: practical tips and trade-offs, three core PromQL functions that work on counters to give you different views of system behavior. Each serves a purpose, and picking the wrong one leads to false positives, missed incidents, or dashboards that confuse your team.

We've all been there: an alert fires at 3 AM, you wake up groggy, and the dashboard shows a spike that's already gone. Or worse, the metric climbs steadily and your alert never triggers because you're watching the wrong function. Today we'll walk through when to use each function and show you the trade-offs so you ship confident monitoring from day one.

Understanding Counter Metrics

Before we dive into rate, irate, and increase, let's be clear about what we're working with. A counter is a metric that only goes up—it never decreases (except during service restarts). Common examples include HTTP request totals, database query counts, errors logged, or bytes transmitted.

Counters are the raw material. You almost never alert directly on a raw counter value because the absolute number grows forever and depends on when your service started. Instead, we use PromQL functions to derive meaningful signals: rates, spikes, trends, and totals over specific time windows.

http_requests_total{job="api-server"} 1847293
http_requests_total{job="api-server"} 1847401  # 108 requests since last scrape
http_requests_total{job="api-server"} 1847512  # 111 requests since last scrape

This counter increments on every request. Raw numbers don't tell you much, but the deltas do. That's where our three functions come in.

rate: Average Per-Second Rate—Use for Alerts

rate() calculates the average per-second rate of change over a time window. It's your go-to for alerting and most dashboards because it smooths out short-term noise and gives you a stable signal.

rate() works by taking the first and last sample in the window, calculating the delta, and dividing by the window duration in seconds. If your scrape interval is 15s and you ask for rate(metric[5m]), Prometheus grabs all samples from the last 5 minutes and computes an average.

# Alert: fire if average request rate exceeds 1000 req/sec over 5 minutes
alert: HighRequestRate
expr: rate(http_requests_total[5m]) > 1000
for: 2m
annotations:
  summary: "API requests averaging {{ $value | humanize }} req/sec"

Why use rate() for alerts? Because it tolerates short spikes. If one scrape interval sees a burst, rate() averages it out. Your alerting system stays calm, and you only fire when there's a sustained problem. Over 5 minutes, even a 10x spike gets diluted into context.

A real scenario: your payment service processes batches every 30 seconds. During batch times, request rate spikes by 2x. With rate(metric[5m]), this daily pattern is invisible—the 5-minute window captures quiet and busy periods together, so the alert threshold is realistic. If you used something more sensitive, you'd be paged every day at batch time.

Trade-off: rate() is slow to respond. If you're using a 5-minute window and traffic suddenly stops, it takes 5 minutes for the alert to fire. The smoothing that makes it stable also makes it lag. Accept this lag for alerting—it's worth the false-positive reduction.

irate: Instant Rate from the Last Two Samples

irate() calculates the instant rate of change using only the last two data points in the window. It's sensitive, responsive, and useful for dashboards where you want to see real-time behavior without averaging.

# Dashboard: show current request rate (updated every scrape interval)
- title: "Requests per Second (Live)"
  expr: irate(http_requests_total[5m])
  
# This uses only the last two samples, so it updates immediately

Because irate() uses only the most recent pair of samples, it reacts instantly to changes. If traffic doubles in one scrape interval, your dashboard updates right away. This makes irate() perfect for observing live behavior during incidents.

Example: during a deploy, you watch your dashboard and spot a sudden drop in request rate using irate()`. You catch it in seconds, not minutes. That immediate feedback lets you rollback fast before impact spreads.

Trade-off: irate() is noisy. A single anomalous sample can spike the value dramatically. If one scrape takes longer than usual, the delta is huge and the rate looks inflated. For this reason, never alert on irate(). Use it only for dashboards and debugging, where human eyes filter out one-off noise.

# Bad: irate is too jittery for alerting
alert: BadAlert
expr: irate(http_requests_total[1m]) > 500  # Don't do this

# Good: rate smooths and is alert-appropriate
alert: GoodAlert
expr: rate(http_requests_total[5m]) > 500  # This is stable

increase: Total Increase Over a Window

increase() returns the absolute increase in the counter value over a time window, expressed in the original units (not per-second). It answers: "how many total events happened in the last N minutes?"

# How many errors in the last hour?
increase(errors_total[1h])

# Alert if more than 1000 errors in the last 5 minutes
alert: HighErrorCount
expr: increase(errors_total[5m]) > 1000
for: 1m

increase() is mathematically equivalent to rate(metric[window]) * window_in_seconds. It's useful when you care about absolute counts, not rates. For instance, billing systems often need "how many jobs completed in the last 24 hours" to compute charges—that's a job for increase().

Another practical use: monitoring disk quota or cache evictions. You might not care about the rate (evictions per second) but rather the total evictions over a day, which tells you whether your cache is thrashing.

# Detect bursty errors by count, not rate
# Useful when 100 errors/sec for 1 second is the same risk as 1 error/sec for 100 seconds
alert: ErrorBurst
expr: increase(errors_total[1m]) > 50
for: 30s

Trade-off: increase() is less sensitive to sustained low rates. If your error rate is 0.5 errors/sec, increase(errors_total[1m]) is only 30 errors—it might not trigger your alert. With rate(), you'd catch it at 0.5 req/sec, which you can threshold independently. Choose based on whether you want to alert on totals or rates.

Common Mistakes

Mistake 1: Using rate() or irate() on gauges. Gauges (like memory usage, temperature, or queue depth) go up and down freely. These functions are meaningless on gauges because the delta is just "the current value minus the previous value," not a rate of change. Use avg(), max(), or deriv() for gauges instead.

Mistake 2: Alerting on irate(). We've said it, but it bears repeating. One delayed scrape, one network hiccup, and your oncall is paged. Always use rate() for alerts.

Mistake 3: Forgetting the time window. rate() and increase() require an explicit window like [5m]. Without it, Prometheus errors. And the window size matters—a 1-minute window is sensitive, a 15-minute window is stable. Pick deliberately based on your alerting tolerance.

Mistake 4: Mixing counter and gauge functions. Don't use rate() on a gauge or avg() on a counter when you really want a rate. It works, but you get nonsensical results and confusion in code reviews.

Mistake 5: Ignoring counter resets. When a service restarts, the counter resets to zero. Prometheus rate() and increase() handle this automatically by detecting the reset and extrapolating. But make sure your monitoring tool is recent enough to support this (Prometheus 2.0+).

Choosing the Right Function: Decision Tree

Are you alerting? Use rate() with a window of 2–10 minutes. Longer windows are more stable; shorter windows catch problems faster. Tune to your tolerance.

Are you building a live dashboard for on-call? Use irate() with any window (the window is mostly ignored). It updates every scrape and shows real-time behavior. Let humans filter the noise.

Do you care about total counts over a period (e.g., jobs completed, errors in a shift)? Use increase(). Pick a window that matches your business logic (1 hour, 1 day, etc.).

Are you working with a gauge? Step back. None of these functions apply. Use aggregators like avg(), max(), p95(), or derivatives.

Key Takeaways

• rate(): Average per-second change over a window. Use for alerts. Stable, slow to respond, forgiving of short spikes.
• irate(): Instant rate from the last two samples. Use for live dashboards and debugging only. Responsive, noisy, never alert on it.
• increase(): Total absolute change over a window. Use when you need raw counts, not rates. Good for quota, billing, and bursty-event detection.
• Always apply these functions to counters, never to gauges. Counters only go up; gauges go up and down.
• Pick your time window deliberately. 5 minutes is a safe default for rate-based alerts. Shorter windows are faster but noisier; longer windows are stable but slow.
• Handle counter resets gracefully. Modern Prometheus does this for you, but know it's happening behind the scenes.

With these three functions in your toolkit and a clear understanding of their trade-offs, you'll write monitoring that actually works: alerts that fire for real problems, dashboards that show what's happening now, and reports that count what matters to your business.