Disk space alerts with predict_linear: a practical guide

Disk space fills up faster than you'd expect—and by the time you notice, your application is already writing to read-only filesystems or crashing. Disk space alerts with predict_linear: a practical guide shows you how to use Prometheus's time-series forecasting to stay ahead of capacity problems. We'll walk through predicting free space hours in advance and triggering alerts before disaster strikes.

Why reactive disk alerts fail

Most teams set up a simple threshold: alert when disk usage exceeds 85%. This approach has a fatal flaw—it reacts too late. On a 1TB disk filling at 100 GB per hour, you go from 85% to 100% in under an hour. By the time you page someone and they start investigating, your database has already stopped accepting writes.

Consider a real scenario: you're running a data pipeline that ingests 50 GB of logs per hour into /var/log. Your monitoring shows disk at 80% at 2 PM. An engineer gets paged, starts their investigation at 2:15 PM, needs 10 minutes to understand the issue, and another 10 minutes to SSH into the box and free up space. By 2:35 PM, the disk is full and queries are failing. Your SLA is already breached.

The real win comes from predicting where your disk will be in 4 hours, not just checking where it is now. Prometheus's predict_linear function lets you do exactly that. Instead of waiting for a threshold to be crossed, you alert when the trajectory will cross it within your window. If you know the disk will be full at 6 PM, you can start mitigation at 2 PM—giving your team actual breathing room.

Predict free space hours ahead

The predict_linear function extrapolates a metric's trend over a specified time range. It fits a line to recent data points and projects where that line will intersect your threshold.

Let's say you're monitoring node_filesystem_avail_bytes across your infrastructure. This metric tells you how many bytes are free on each mounted filesystem. A typical query looks like this:

node_filesystem_avail_bytes{mountpoint="/data",fstype!="tmpfs"}

To predict where free space will be in 4 hours, use:

predict_linear(node_filesystem_avail_bytes{mountpoint="/data",fstype!="tmpfs"}[1h], 4 * 3600)

Breaking this down: we take the last hour of data [1h] and fit a linear regression to it. The second argument, 4 * 3600, is 4 hours in seconds. The result is a projected byte count for where that filesystem will be 4 hours from now.

Why 1 hour of lookback and 4 hours of prediction? In production, 1 hour gives you enough data points to smooth out noise without being stale. If you use only 5 minutes, a temporary spike in I/O will skew your prediction wildly. With 1 hour, you're capturing actual growth trends. Four hours is typically enough runway to drain a disk safely—you've got time to add capacity, delete logs, or migrate workloads before things get critical. If you set it to 24 hours, the prediction becomes nearly useless because too many variables can change the growth rate.

In practice, you might need to adjust these windows based on your environment. A Kafka broker ingesting events might need a 2-hour lookback and 2-hour prediction window. A database server with stable write patterns might work fine with 1 hour and 6 hours. Start conservative and adjust based on what you observe.

Alert when projected free space hits zero within 4 hours

Now we build the alert rule. We want to fire an alert if the predicted free space is less than 0 bytes—meaning the disk will be full within our 4-hour window.

groups:
  - name: disk_space
    interval: 30s
    rules:
      - alert: DiskWillBeFullIn4Hours
        expr: predict_linear(node_filesystem_avail_bytes{fstype!="tmpfs",mountpoint!~"^/dev|^/sys|^/proc|^/run"}[1h], 4 * 3600) < 0
        for: 5m
        annotations:
          summary: "Disk {{ $labels.device }} on {{ $labels.instance }} will be full in ~4 hours"
          description: "Predicted free space: {{ $value | humanize }}B. Current free: {{ with query \"node_filesystem_avail_bytes{device=\\\"{{ $labels.device }}\\\"}\" }}{{ . | first | value | humanize }}B{{ end }}"
        labels:
          severity: warning
          team: infrastructure

The for: 5m clause prevents flapping—we only alert if the condition holds for 5 consecutive minutes. This smooths out temporary read bursts that might trigger a false positive. Without it, a sudden write spike followed by a deletion would create an alert that fires and clears every few minutes, exhausting your team.

The filter fstype!="tmpfs" excludes temporary filesystems like /run and /dev/shm, which are typically transient and less concerning. The mountpoint!~ regex further excludes system mounts you probably don't care about—these are kernel-managed and rarely fill up in a way you can act on.

In your alerting backend, ensure this alert routes to the infrastructure team immediately. Many teams have this alert wake someone up during business hours but only page on-call during nights—you don't want to be blindsided at 3 AM with a full disk because the alert was deprioritized.

Combine with a percentage threshold

Prediction alone isn't always enough. A 10TB SAN running at 95% full won't hit zero in 4 hours, but it's already in dangerous territory. Additionally, if Prometheus scrapes are inconsistent or there's a data gap, predict_linear can give unreliable results. Combine your predictive alert with a percentage-based alert for situations where disks are already critically full and as a safety net.

      - alert: DiskPercentageCritical
        expr: (node_filesystem_avail_bytes{fstype!="tmpfs",mountpoint!~"^/dev|^/sys|^/proc|^/run"} / node_filesystem_size_bytes) < 0.05
        for: 2m
        annotations:
          summary: "Disk {{ $labels.device }} on {{ $labels.instance }} is only {{ $value | humanizePercentage }} free"
          description: "This disk is critically full. Investigate immediately. Free: {{ $value | humanize }}B."
        labels:
          severity: critical
          team: infrastructure

      - alert: DiskPercentageWarning
        expr: (node_filesystem_avail_bytes{fstype!="tmpfs",mountpoint!~"^/dev|^/sys|^/proc|^/run"} / node_filesystem_size_bytes) < 0.15
        for: 5m
        annotations:
          summary: "Disk {{ $labels.device }} on {{ $labels.instance }} is {{ $value | humanizePercentage }} full"
          description: "Current free space: {{ $value | humanize }}B. Monitor growth rate and plan accordingly."
        labels:
          severity: warning
          team: infrastructure

Here's the rationale: the 5% critical threshold catches cases where you're already in trouble and need immediate action. The 15% warning threshold gives you room to breathe while still flagging concerning trends. The for: 2m on critical means you need two consecutive minutes of less than 5% free before we page you—this avoids alerts during temporary spikes but still catches real problems quickly. For the warning, for: 5m is reasonable because you have a bit more time to respond.

Real example: your application experiences a 10-minute batch job that writes 50 GB to disk, then deletes it all. Without the for clause, this triggers your critical alert every time. With for: 2m on critical, you only alert if the disk stays above 95% full for a full 2 minutes—the batch job completes in 10 minutes but doesn't stay at 95% the whole time, so no false alert.

Cover every mounted filesystem

The queries above use label-based filtering to apply the same logic to all filesystems. But you'll likely have different thresholds for different mounts. A root filesystem should probably be monitored more aggressively than a log volume, which can tolerate filling up. Your Kubernetes node might have /, /var/lib/kubelet, and several pod volumes—each with different growth patterns and recovery strategies.

Use Prometheus's label matching to apply specific rules:

      - alert: RootFilesystemWillBeFull
        expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/",fstype!="tmpfs"}[1h], 4 * 3600) < 0
        for: 5m
        annotations:
          summary: "Root filesystem on {{ $labels.instance }} will be full in 4 hours"
          description: "Predicted free space: {{ $value | humanize }}B. This affects system stability."
        labels:
          severity: critical

      - alert: DataFilesystemWillBeFull
        expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/data",fstype!="tmpfs"}[1h], 4 * 3600) < 0
        for: 5m
        annotations:
          summary: "Data filesystem on {{ $labels.instance }} will be full in 4 hours"
          description: "Database writes may fail. Predicted free space: {{ $value | humanize }}B"
        labels:
          severity: critical

      - alert: LogFilesystemWillBeFull
        expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/var/log",fstype!="tmpfs"}[1h], 4 * 3600) < 0
        for: 5m
        annotations:
          summary: "Log filesystem on {{ $labels.instance }} will be full in 4 hours"
          description: "Check log rotation policies and consider archiving old logs. Predicted free space: {{ $value | humanize }}B"
        labels:
          severity: warning

      - alert: ContainerVolumeWillBeFull
        expr: predict_linear(node_filesystem_avail_bytes{mountpoint=~"/var/lib/docker.*",fstype!="tmpfs"}[1h], 4 * 3600) < 0
        for: 5m
        annotations:
          summary: "Container volume {{ $labels.mountpoint }} on {{ $labels.instance }} will be full in 4 hours"
          description: "Container runtime may fail to write. Free up container storage or prune unused images."
        labels:
          severity: warning

Root and data filesystems get critical severity because application writes will fail. Log filesystems get warning severity because you can usually clear them—though you still want to know about it. Container volumes get warning because you can prune images and stopped containers to free space relatively quickly.

In Grafana, create a dashboard panel to visualize the predicted trajectory alongside actual usage. This helps your team understand the health of each mount and plan capacity before alerts fire. A simple panel showing predict_linear(...), current node_filesystem_avail_bytes, and a horizontal line at zero gives everyone on the team visibility into which disks are at risk.

Common mistakes

• Using too short a lookback window. A 5-minute [5m] lookback is too noisy. Use at least 1 hour so the linear regression has enough data to be meaningful. A single I/O spike in 5 minutes will make the prediction useless.
• Setting prediction windows too long. A 24-hour prediction is almost meaningless—too many unknowns. Stick to 2–6 hours depending on your disk growth patterns. Beyond that, you're guessing.
• Ignoring all filesystems. Don't just monitor / and /data. Check mounted NFS shares, block device mounts, container volumes, and any external storage. They fill up just as fast and are often harder to recover from.
• Forgetting the for clause. Without for: 5m, you'll get flapping alerts during every temporary I/O burst. Require sustained high usage before alerting. This is especially important for workloads with bursty patterns.
• Not accounting for node resets and data gaps. If a Prometheus scrape fails or there's a gap in data, predict_linear may give nonsensical results. If your instance reboots, the metric resets. Always pair predictions with percentage thresholds as a safety net.
• Alerting on filesystems that don't matter. Tmpfs and proc filesystems don't need alerts. Use filters to focus on real storage you can actually act on. Every alert that fires should require real action.
• Not tuning for your actual growth patterns. If your logs rotate hourly, a 4-hour prediction might be too long. If your data grows slowly, you might need 12 hours. Observe your actual growth rates and tune accordingly.

Key takeaways

• predict_linear lets you project where free disk space will be hours ahead, giving your team actionable lead time instead of reacting to emergencies.
• Combine predictive alerts (will disk hit zero?) with percentage-based alerts (is disk already critically full?) for defense in depth and to handle edge cases where prediction is unreliable.
• Use 1-hour lookback and 4-hour prediction windows as a starting point; tune based on your disk growth rates and team response times.
• Apply the same alert rules across all relevant mountpoints using label matching, but tailor severity and for duration based on how critical each filesystem is.
• Always include the for clause to avoid false positives from transient I/O spikes and temporary writes.
• Filter out tmpfs and system filesystems to keep alert noise down and focus on storage you can actually manage and act on.
• Monitor real growth over several weeks to refine your lookback and prediction windows for your specific workload patterns.