Metrics for capacity planning for Monitoring developers

Metrics for capacity planning for Monitoring developers

Running out of disk space at 3 AM, or watching your database CPU spike to 100% during the weekly sync—these scenarios become preventable when you instrument your systems with the right capacity metrics. As monitoring developers, we often focus on alerting on immediate problems, yet the equally critical skill is forecasting future resource constraints before they become incidents. This article walks through the practical approach of collecting, analyzing, and acting on utilization trends to keep your infrastructure scaling smoothly.

Why capacity planning matters in monitoring

Capacity planning isn't glamorous, but it's the difference between a predictable scaling roadmap and reactive firefighting. When we talk about metrics for capacity planning for Monitoring developers, we're discussing how to extract signal from historical data and make confident predictions about when you'll need to add resources.

The core challenge: most teams run monitoring that alerts on thresholds—"CPU is above 80%"—but they lack a systematic view of utilization trends. You might have a spike every Friday at 2 PM that clears by 3 PM, or you might have a slow creep upward that means you'll saturate your storage in six weeks. Without quantifying these patterns, you're making infrastructure decisions on gut feel rather than data.

Prometheus and Grafana give us the tools to build this visibility. The key is knowing which metrics to collect and how to interpret them.

Track utilization trends

The foundation of capacity planning is understanding how your resources are being used over time. This isn't a single number; it's a story that unfolds across weeks and months.

What to measure

Start with the essentials:

• CPU utilization (user, system, iowait)
• Memory usage (active, cached, available)
• Disk I/O (bytes read/written per second)
• Network throughput (inbound and outbound bytes per second)
• Database connections (active, idle, waiting)
• Queue depth (for job queues or task backlogs)
• Storage utilization (bytes used, bytes available)

For a typical SaaS backend, we instrument application-level metrics as well: request latency percentiles, error rates by endpoint, and cache hit rates. These correlate with resource consumption and help you understand why a metric is trending up.

Collecting utilization data

Your Prometheus scrape configuration should capture node metrics at regular intervals—60 seconds is a standard starting point. Here's a typical setup:

global:
  scrape_interval: 60s
  evaluation_interval: 60s

scrape_configs:
  - job_name: 'node'
    static_configs:
      - targets: ['localhost:9100']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'node_(cpu|memory|disk|network).*'
        action: keep
  - job_name: 'postgres'
    static_configs:
      - targets: ['localhost:9187']
    scrape_interval: 30s
  - job_name: 'app'
    static_configs:
      - targets: ['localhost:8080']

The `metric_relabel_configs` above keeps only the metrics relevant to capacity planning, reducing cardinality and storage overhead. For long-term trend analysis, set up a retention policy of at least 6 months—13 weeks minimum—so you can see seasonal patterns.

Visualizing trends in Grafana

Create a dashboard that shows utilization over multiple time windows. A practical layout includes:

• 7-day view: spot daily cycles and weekly patterns
• 30-day view: catch gradual trends and anomalies
• 90-day view: see the macro picture for quarterly planning

A sample Prometheus query to calculate 30-day average memory usage:

avg_over_time(node_memory_MemAvailable_bytes[30d])

And the 95th percentile over the same window:

histogram_quantile(0.95, sum(rate(node_memory_MemAvailable_bytes[30d])) by (instance))

The difference between average and peak tells you a lot. If your average CPU is 30% but the 95th percentile is 75%, you have bursty workloads that need careful handling during scaling decisions.

Forecast with predict_linear

Once you have months of utilization data, the next step is asking: "When will we run out?" Prometheus provides the predict_linear() function to forecast resource exhaustion based on linear regression.

How predict_linear works

The function fits a line to your metric over a specified time window and extrapolates forward. The syntax is:

predict_linear(metric[time_window], seconds_ahead)

For example, to forecast disk usage 30 days from now based on the last 7 days of data:

predict_linear(node_filesystem_avail_bytes{mountpoint="/"}[7d], 30 * 24 * 3600)

This is powerful but has limitations. It assumes the trend continues linearly, which rarely holds in production. A slow creep might suddenly accelerate after a product launch, or plateau after you optimize a query.

Practical forecasting workflow

We use predict_linear() as a flag, not gospel. Here's the workflow:

1. Calculate the predicted value 60 days out for each major resource
2. If prediction suggests 85% utilization within 60 days, create a ticket for capacity planning
3. Cross-check predictions against application roadmap (are you shipping features that will increase load?)
4. Adjust for known events (holiday traffic dips, planned migrations, new customer onboarding)

A practical Grafana alert rule:

groups:
  - name: capacity
    interval: 1h
    rules:
      - alert: DiskFullIn60Days
        expr: predict_linear(node_filesystem_avail_bytes{mountpoint="/var/lib/postgresql"}[7d], 60 * 24 * 3600) < 1099511627776
        for: 10m
        annotations:
          summary: "PostgreSQL disk may be full in ~60 days"
          description: "Current trend: {{ $value | humanize }}B available in 60 days"

The threshold (1 TB) gives you buffer time to order hardware or implement archival before the crisis hits. Adjust based on your tolerance for risk and lead time for procurement.

Going beyond predict_linear

If your business is growing fast, linear models break down. Consider using Prometheus with external time-series forecasting tools—some teams export metrics to Python with Prophet or use VictorOps-style forecasting. But for most teams, predict_linear() combined with quarterly reviews is sufficient.

Peak vs average load

A critical distinction in capacity planning: your infrastructure must handle peak load, not average load. But many teams size based on average, then get surprised by outages.

Understanding the gap

Let's say your PostgreSQL database averages 40 connections but peaks at 200 connections on Friday afternoons. If you only scale to handle 60 average connections, you'll hit connection limits and start rejecting queries. The peak determines your SLA; the average determines your cost efficiency.

Calculate this explicitly:

# Average over the last 30 days
avg_load_30d: avg_over_time(pg_stat_activity_count[30d])

# 99th percentile—your effective peak
peak_load_30d: histogram_quantile(0.99, sum(rate(pg_stat_activity_count[30d])) by (instance))

# Capacity headroom (e.g., if max_connections=300, headroom = (300 - peak) / peak)

A dashboard showing both simultaneously makes the story clear. We typically aim to keep headroom of at least 20% above the 99th percentile load, so occasional spikes don't saturate the system.

Seasonal patterns and business events

E-commerce platforms see 10x peak during Black Friday. SaaS products spike when campaigns launch. Data pipeline workloads have predictable monthly patterns (month-end reporting). Overlay these events with your historical metrics to set realistic peaks.

Create a Grafana variable that marks known events:

annotations:
  - name: Business Events
    datasource: Prometheus
    iconColor: 'blue'
    tags: ['capacity']

Then manually annotate your graph with "Black Friday 2024-11-29" or "Migration window 2024-12-15" so capacity decisions account for non-linear spikes.

Review quarterly

Capacity planning isn't a one-time task; it's a quarterly ritual that keeps your infrastructure aligned with business growth.

Setting up the review cadence

Schedule a 60-minute meeting each quarter with your platform/infrastructure team and a representative from product:

• Week 1: pull utilization reports from the previous quarter
• Week 2: review forecasts for the next quarter and identify risks
• Week 3: size new instances or services needed; create procurement tickets
• Week 4: execute purchases, schedule deployments

This gives you a 4-6 week lead time before new resources are needed, which usually aligns with hardware delivery and integration timelines.

What to review

For each major resource (CPU, memory, storage, network, database connections), capture:

• Utilization at the 50th, 95th, and 99th percentiles for the past quarter
• Growth rate (% increase month-over-month)
• Forecast for the next 90 days
• Any incidents related to capacity (e.g., "lost 30 min to a cache miss cascade that spiked CPU to 95%")
• Planned changes in the next quarter (new services, sunset of legacy systems, marketing campaigns)

A simple template in your team wiki:

Q1 2025 Capacity Review
Date: 2025-03-28

CPU:
  Q4 2024 95th percentile: 62%
  Q4 2024 growth rate: +8% MoM
  Q1 2025 forecast (90d): 68%
  Action: None, headroom sufficient

Memory:
  Q4 2024 95th percentile: 78%
  Q4 2024 growth rate: +12% MoM
  Q1 2025 forecast (90d): 92%
  Action: Order 128GB additional RAM for prod-db-03, deploy in May

Storage (/var/lib/postgresql):
  Q4 2024 95th percentile: 71%
  Q4 2024 growth rate: +5% MoM
  Q1 2025 forecast (90d): 79%
  Action: Implement table partitioning for events table (Q2 project); defer hardware until Q3 if partitioning successful

Automation in quarterly reviews

Don't manually pull these numbers. Build a Grafana dashboard that auto-generates a PDF or Slack summary at the end of each quarter:

#!/bin/bash
# Run via cron on the last Friday of each quarter

PERIOD_END=$(date +%Y-%m-%d)
PERIOD_START=$(date -d '90 days ago' +%Y-%m-%d)

# Query Prometheus API for key metrics
curl -s "http://prometheus:9090/api/v1/query_range?query=avg_over_time(node_cpu_usage[90d])&start=${PERIOD_START}&end=${PERIOD_END}&step=86400" | jq '.data.result' > cpu_report.json

# Generate markdown and post to Slack
echo "Q$((($(date +%m)-1)/3+1)) $(date +%Y) Capacity Report" > capacity_report.md
# ... append metrics from JSON
slack_post --channel "#platform" --file capacity_report.md

Common mistakes

Over-relying on predict_linear without domain knowledge: Linear forecasts assume the past repeats. If you're about to launch a feature that increases traffic 5x, the model will be wildly off. Always feed business context into capacity decisions.

Sizing for average instead of peak: We've seen teams get this backwards and end up scrambling mid-incident. The 99th percentile (or worst-case peak you've observed) is your sizing target.

Ignoring network and I/O in favor of CPU and memory: CPU and memory are visible, but network saturation and disk I/O can be sneaky bottlenecks. A fully utilized NIC or a disk hitting its IOPS limit will degrade your system long before CPU maxes out.

Collecting metrics but not acting on them: A dashboard that no one reviews is just noise. Tie capacity reviews to calendar events and empower someone to own the process.

Not accounting for variability in predictions: If predict_linear shows you'll be at 87% in 60 days, don't schedule capacity exactly at 60 days—add buffer. Forecasts are estimates, not guarantees.

Key takeaways

• Capacity planning is data-driven prediction, not guesswork. Collect utilization trends for CPU, memory, disk, network, and application-specific metrics over at least 6 months.
• Use predict_linear() to forecast resource exhaustion, but treat it as a signal to review, not as ground truth. Combine predictions with business context and known events.
• Size infrastructure for peak load (95th or 99th percentile), not average load. The gap between peak and average determines how efficiently you run.
• Establish a quarterly capacity review cadence with clear ownership. Review historical trends, update forecasts, and create procurement tickets with 4-6 week lead time.
• Automate metric extraction and reporting where possible. Hand-rolled spreadsheets become stale; dashboards queried programmatically stay fresh.
• Build in headroom (20% above predicted peak is a reasonable target) to absorb unexpected spikes and avoid capacity-related incidents.