Retention and storage planning for Monitoring developers

Retention and storage planning for Monitoring developers

Prometheus keeps your metrics in memory and on disk, but storage isn't infinite. By day 53 of your monitoring journey, you've likely noticed that your disk usage grows steadily as new samples arrive—and without a retention strategy, you'll either run out of space or spend a fortune on cloud storage. The good news is that retention and storage planning for monitoring developers isn't complex once you understand the trade-offs between keeping raw data, downsampling, and archiving to long-term storage.

Default retention is 15 days

Prometheus ships with a default retention window of 15 days. This means that after 15 days, older samples are deleted. For many small deployments, this works fine—you catch incidents quickly, investigate for a couple of weeks, and move on. But 15 days isn't always enough.

Consider a production SaaS platform where you need to correlate a customer's poor experience from last week with a deployment that went out five days ago. Or imagine a trading system where compliance requires you to retain audit trails of all metric changes for 90 days. In these cases, the default window is far too short.

You configure retention using the --storage.tsdb.retention.time flag when starting Prometheus:

./prometheus \
  --config.file=prometheus.yml \
  --storage.tsdb.path=/var/lib/prometheus \
  --storage.tsdb.retention.time=90d

You can also set retention by size instead of time using --storage.tsdb.retention.size. This is useful in containerized environments where you have a fixed disk quota. If you set both, Prometheus will honor whichever limit is hit first:

./prometheus \
  --storage.tsdb.retention.size=50GB \
  --storage.tsdb.retention.time=90d

The trade-off is straightforward: longer retention requires more disk space. Before you extend retention to 6 months, you need to know how much storage you'll actually consume.

Estimate bytes per sample

To plan your storage, you need to understand how many bytes each metric sample consumes. Prometheus doesn't store metrics as human-readable JSON; it uses a compressed binary format. A typical sample costs about 1-2 bytes on disk after compression, though this varies based on your time-series cardinality and how compacted your blocks are.

Here's a practical formula:

Suppose you're scraping 10,000 active time series every 60 seconds. Here's the math:

• 10,000 time series × 60 samples/hour × 24 hours × 90 days = 129.6 million samples
• 129.6 million × 1.5 bytes (average compressed size) ≈ 194 GB

Add a 20% buffer for WAL (write-ahead log) and metadata, and you're looking at roughly 232 GB for a 90-day retention window.

You can verify this empirically by checking your Prometheus data directory:

du -sh /var/lib/prometheus/

Then divide by your current retention days to get a daily burn rate. If you're using 1 GB per day with 10,000 series at a 60-second interval, you can forecast that extending retention to 180 days will cost you 180 GB.

In our experience across e-commerce and SaaS deployments, active time-series count tends to grow over time as teams add new instrumentation. Estimate high: assume your cardinality will double in 12 months, and plan your storage ceiling accordingly.

Downsample old data

Not all historical data needs to be stored at full resolution. A month-old metric doesn't need to be sampled every 60 seconds; you can afford to look at it at 5-minute granularity. This is where downsampling comes in.

Prometheus doesn't natively downsample, but you can use Thanos or similar tools to aggregate old data and reduce storage. The pattern is simple:

• Keep raw data at 1-minute resolution for the last 7 days (hot data)
• Downsample to 5-minute resolution for days 8–30
• Downsample to 1-hour resolution for days 31–180

This approach cuts your storage footprint to roughly 30% of what you'd need at full resolution, while preserving enough fidelity to detect trends and correlate incidents from weeks ago.

If you're using Thanos, you configure downsampling rules in its compactor component. Here's a sample configuration:

compaction:
  blocks-dir: /var/thanos/blocks
  consistency-delay: 30m
  retention:
    resolution-raw: 30d
    resolution-5m: 90d
    resolution-1h: 180d
  
  downsampling:
    disable: false

With this setup, Thanos automatically creates downsampled versions of your blocks. Your Prometheus instance stores raw data; Thanos handles the rest. When you query historical data in Grafana, Thanos transparently returns downsampled blocks if full-resolution data isn't available.

The trade-off is latency: querying downsampled data takes slightly longer, and you lose precision. But for dashboards showing CPU usage over the past month, a 5-minute resolution is perfectly adequate, and the storage savings are substantial.

Move long-term data to object storage

For deployments that require months or years of data, keeping everything on local disk becomes prohibitively expensive. This is where moving to object storage—S3, GCS, or Azure Blob Storage—makes sense.

Thanos integrates seamlessly with object storage. Instead of keeping blocks on a local filesystem, you configure Thanos to upload them to S3. This gives you several advantages:

• Cost: S3 Standard costs roughly $0.023 per GB per month. Local NVMe on cloud instances costs 5–10x more.
• Durability: S3 provides 11 nines of durability. Your local disk is a single point of failure.
• Scalability: You can retain years of data without worrying about disk capacity on your Prometheus instance.
• Query performance: Thanos caches frequently-accessed blocks locally, so recent queries are fast. Older queries hit S3, which is slower but acceptable for historical analysis.

Here's a Thanos sidecar and object storage configuration:

sidecar:
  prometheus:
    url: http://localhost:9090
  object-store:
    type: S3
    config:
      bucket: my-monitoring-bucket
      endpoint: s3.amazonaws.com
      access_key: xxx
      secret_key: xxx
      
blocks-dir: /var/thanos/blocks
objstore-config:
  type: S3
  config:
    bucket: my-monitoring-bucket
    endpoint: s3.amazonaws.com
    access_key: xxx
    secret_key: xxx

The sidecar runs alongside your Prometheus instance and uploads completed TSDB blocks to S3. A Thanos querier can then hit multiple sidecars, fetch blocks from S3, and merge results. Your Prometheus instance itself only needs enough local disk for the current retention window (typically 2–7 days); older data lives in object storage.

In production trading and ride-hailing systems we've deployed, this architecture has been essential. You can afford to keep 3 years of metrics because storage is cheap, yet queries for recent data remain fast because they hit local blocks first.

Common mistakes

Setting retention too high without capacity planning: We've seen teams increase --storage.tsdb.retention.time to 180 days, then run out of disk within a week because they didn't account for cardinality growth. Always estimate conservatively and monitor actual usage.

Forgetting about the WAL: The write-ahead log consumes 5–10% of your total disk usage. If you're planning for exactly 100 GB of metrics, your actual footprint will be 110 GB. Leave headroom.

Downsampling everything immediately: It's tempting to downsample all data after 24 hours to save space. But dashboards and alerting rules written against 60-second granularity will break. Downsample gradually: keep full resolution for at least 7 days, then step down.

Using object storage without a caching layer: If you query S3-backed metrics directly without Thanos querier caching, every query becomes slow. Ensure your architecture includes a caching layer or local hot storage.

Key takeaways

• Prometheus defaults to 15 days of retention; extend it based on your incident investigation and compliance needs, but monitor disk usage carefully.
• Estimate storage using the formula: time-series count × samples per day × retention days × 1.5 bytes per sample, then add 20% for overhead.
• Downsample old data to 5-minute or hourly resolution using Thanos or similar tools to cut storage costs by 60–70% without losing analytical value.
• Move data older than 30 days to S3 or equivalent object storage; the cost per GB drops dramatically, and Thanos caching keeps queries fast.
• Plan for cardinality growth; your metrics will likely double in 12 months, so provision storage generously and revisit your strategy quarterly.