Anomaly detection with PromQL for Monitoring developers

Anomaly detection with PromQL for Monitoring developers

Your monitoring system sends alerts at 2 AM for a metric that always spikes on Tuesday mornings—false positives drain your team's energy and trust in alerting. Anomaly detection with PromQL for Monitoring developers gives you tools to catch genuinely unusual behavior while filtering out predictable patterns. By shifting from rigid thresholds to adaptive baselines, you'll spend less time chasing noise and more time responding to real incidents.

Why fixed thresholds fail you

Static thresholds work until they don't. A database query that normally takes 50ms during business hours might consistently hit 100ms after midnight batch jobs start. Your CPU utilization might run at 65% on weekdays but only 20% on Sundays. Setting a single alert threshold catches neither the real problem (Wednesday's unexpected spike to 150ms) nor the expected Tuesday maintenance window.

Most teams discover this the hard way: dozens of alerts during predictable load patterns, no alerts when something actually breaks. Anomaly detection with PromQL moves beyond "is this value too high" to "is this value unusual compared to what we normally see."

Compare to last week—offset for seasonal baselines

The simplest form of anomaly detection compares current behavior to the same time period from previous weeks. This captures day-of-week seasonality and accounts for predictable business cycles without needing complex statistical models.

The offset modifier in PromQL lets you retrieve historical data points. By comparing this week's 9 AM CPU against last week's 9 AM CPU, you're measuring apples to apples:

- alert: UnusualCPUSpike
  expr: |
    (rate(node_cpu_seconds_total[5m]) - ignoring(le) offset(rate(node_cpu_seconds_total[5m] offset 1w), 5m))
    / ignoring(le) offset(rate(node_cpu_seconds_total[5m] offset 1w), 5m)
    > 0.25
  for: 10m
  annotations:
    summary: "CPU is 25% higher than last week at this time"

This query calculates the percentage difference between current CPU and last week's CPU at the same time of day. If last Tuesday at 9 AM you used 40% CPU and this Tuesday at 9 AM you're using 55%, that's a 37.5% increase—worth investigating.

In practice, offset comparisons work well for:

• Day-of-week patterns: Comparing Tuesday to Tuesday, not Tuesday to Wednesday
• Recurring batch jobs: Known weekend backups or nightly maintenance
• Business hours variance: 9 AM peak traffic versus 3 AM quiet hours

The limitation: you need at least a week of historical data, and large one-time events last week get baked into your baseline. If last Tuesday had a production incident that caused CPU to spike, this Tuesday's normal traffic looks like an anomaly.

Z-scores—standard deviation over time windows

When you need to handle genuinely new patterns, z-scores measure how far a value deviates from the recent average, expressed in standard deviations. A z-score of 2 means "this is two standard deviations away from normal," which statistically occurs in only 5% of observations.

PromQL doesn't have built-in z-score functions, but you can calculate them by combining aggregation functions over time windows. Here's the pattern for detecting unusual request latency:

- alert: UnusualLatencySpike
  expr: |
    (
      histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))
      -
      avg_over_time(histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))[1h:5m])
    )
    /
    stddev_over_time(histogram_quantile(0.95, rate(http_request_duration_seconds_bucket[5m]))[1h:5m])
    > 2
  for: 5m
  annotations:
    summary: "p95 latency is {{ $value | humanize }}σ above normal"

Breaking this down:

• histogram_quantile(0.95, ...) gets your 95th percentile latency
• avg_over_time(...[1h:5m]) calculates the mean latency over the past hour, evaluated every 5 minutes
• stddev_over_time(...[1h:5m]) calculates how much latency normally varies
• The division gives you z-scores; when this exceeds 2, you're in unusual territory

Z-score detection shines for:

• Real-time anomalies: No historical week needed, works on fresh metrics
• Multiple baselines: Different services have different normal variability
• Gradual changes: Catches slow degradation (average latency drifting up) separately from sudden spikes

The downside: a one-hour window is still arbitrary. If your entire fleet gradually starts returning slower responses, your z-score baseline shifts with it and won't alert. You're flying blind to slow, fleet-wide degradation.

predict_linear—forecast disk full times

Some anomalies aren't about "unusual now" but about "unusual if the current trend continues." Disk space is the classic example: you don't care that your disk usage rate is 5 MB/min, but if it keeps growing at that rate for the next 8 hours, you'll run out of space.

The predict_linear function extrapolates metric trends and predicts future values. This lets you alert before the problem occurs, not after:

- alert: DiskWillFillIn8Hours
  expr: |
    predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[1h], 8 * 3600)
    < 5368709120
  for: 5m
  annotations:
    summary: "Disk {{ $labels.device }} will fill in approximately 8 hours"
    description: "Current available space: {{ (node_filesystem_avail_bytes / 1024 / 1024 / 1024) | humanize }}GB"

This query looks at how available disk space has changed over the past hour, projects that trend forward 8 hours (28800 seconds), and alerts if predicted available space drops below 5 GB.

The math: if you have 100 GB free and you're using disk at 2 GB/hour, predict_linear says "in 8 hours you'll have 84 GB free," which is fine. But if you have 100 GB free and you're using disk at 15 GB/hour, predict_linear says "in 8 hours you'll need 20 GB but only have 20 GB"—time to alert and investigate.

Where predict_linear works well:

• Capacity exhaustion: Disk, memory, connection pools
• Linear growth patterns: Log files, temporary data accumulation
• Degrading hardware: When error rates or latency drift consistently upward

Its limitation: it assumes the trend stays linear. A disk that's been filling at 10 GB/day might suddenly fill at 100 GB/day when a logging misconfiguration multiplies output volume. The prediction becomes useless. Use predict_linear as one signal, not your only defense.

Keep it simple—start with static thresholds that work

Before you build a sophisticated anomaly detection pipeline, ship something that actually fires on real problems. Static thresholds, despite their limitations, have one enormous advantage: they work immediately and everyone understands them.

A good approach:

1. Start with obvious static thresholds. Alert when CPU exceeds 90%, memory exceeds 85%, error rate exceeds 5%, p95 latency exceeds 2 seconds. These catch the big fires.
2. Run these in production for a week. Track how many fire, which are true positives, which are noise. Measure on-call fatigue.
3. Add offset comparisons for high-noise metrics. If your database CPU always spikes at 9 AM but last year's 9 AM also spiked, compare week-to-week instead of using a flat threshold.
4. Graduate to z-scores only for metrics where you have good data. After two months of clean baseline, not day one.

Many teams skip step 1 and jump to complex models, only to have those models fire every weekend or miss obvious problems because the baseline drifted. Static thresholds are unsexy but reliable. Wrap them in a framework where you can add sophistication without removing safety.

Common mistakes

• Using too short a time window for baselines. A 15-minute window for z-scores captures noise, not signal. Use at least 1 hour; 4-6 hours is better for metrics with visible patterns.
• Forgetting that offset data doesn't exist for new deployments. A fresh Prometheus or a newly restarted exporter won't have week-old data. Your offset queries return empty results. Fall back to static thresholds gracefully.
• Over-alerting on percent changes. If last week your error rate was 0.01% and this week it's 0.02%, that's a 100% increase but still nothing to page on. Use absolute thresholds plus percent thresholds together.
• Setting z-score thresholds too low. A z-score of 1.5 fires on events that occur roughly 7% of the time—you'll get alerts constantly. Start at 2 (5%) or 3 (0.1%) and tune from there.
• Applying predict_linear to metrics with bursty behavior. Predicting from one hour of a metric that spikes unpredictably gives you garbage predictions. Use it only for smooth, monotonic trends.

Key takeaways

• Anomaly detection with PromQL for Monitoring developers starts simple: offset comparisons for day-of-week seasonality catch the easiest false positives.
• Z-scores measure deviation from recent averages and work well when you need real-time detection without historical week-long baselines, but use 1+ hour windows to avoid noise.
• predict_linear forecasts capacity exhaustion and gives you advance warning on linear trends, but fails on bursty or sudden changes.
• Static thresholds are your foundation—they're reliable, predictable, and everyone understands them. Add sophistication incrementally after proving you need it.
• No single approach catches everything; combine multiple signals and always include absolute thresholds as a safety net underneath your statistical models.