Monitoring costs: a practical guide for developers

The cost-observability tradeoff

Observability transforms how we build and operate systems—until the bill arrives. Every metric series ingested, every trace collected, every log line indexed adds up quickly, and many teams discover too late that their monitoring infrastructure costs more than their production workloads. Monitoring costs: a practical guide for developers walks through the levers we actually control to keep observability valuable without burning budget.

We've all watched a single cardinality explosion or poorly-configured retention policy spiral costs from a few hundred to several thousand dollars per month in weeks. The good news: most of these problems have straightforward solutions that don't require abandoning observability. They require being intentional about what we collect, how long we keep it, and which dashboards we actually use.

Cardinality — Fewer series, lower cost

Cardinality is the number of unique time series your monitoring system stores. A metric with low cardinality might have 10 series; one with high cardinality could have millions. Most monitoring platforms charge by ingested volume or series count, so cardinality is often your biggest cost lever.

Consider a typical web service metric:

http_request_duration_seconds{
  method="GET",
  path="/api/orders",
  status="200",
  region="us-east-1",
  service="order-service",
  instance="pod-1"
}

That single metric definition creates one series. Now add unbounded labels:

http_request_duration_seconds{
  method="GET",
  path="/api/orders/{id}",    # Bad: unique per request ID
  status="200",
  user_id="12345",            # Bad: unique per user
  region="us-east-1"
}

If you have 10,000 users and 1,000 unique request paths per hour, you've created 10 million series from one metric. Your bill explodes.

High-cardinality traps we see repeatedly:

• Request IDs or trace IDs as labels — Every request gets a unique ID. Never label with it.
• User IDs or customer IDs — Scales with your user base, not your infrastructure.
• Timestamps or sequence numbers — Creates infinite series.
• Database query text — Each unique query is a new series.
• IP addresses — Scales with traffic volume.

The fix is ruthless label discipline. Keep only labels that have bounded values and represent meaningful operational dimensions:

http_request_duration_seconds{
  method="GET",
  endpoint="/api/orders",     # Bounded: ~50-200 endpoints
  status="200",
  region="us-east-1",         # Bounded: 3-5 regions
  service="order-service"     # Bounded: your services
}

For anything that needs to be high-cardinality (like customer ID), store it in logs or traces instead. They handle cardinality differently and cost less per item.

A practical audit: export your top 50 metrics by series count from Prometheus. Most teams find 2-3 metrics generating 60% of their cardinality. Often it's application instrumentation that was never reviewed in production. Fixing those three metrics can cut your bill by half.

Retention — Keep raw data briefly, downsample the rest

Raw metrics at full resolution are expensive to store. You don't need 15-second granularity from three months ago. Most teams can optimize retention dramatically by keeping recent data raw and downsampling older data to lower resolution.

A typical strategy:

• Raw 15-second resolution: 7-14 days (operational debugging window)
• 1-minute aggregates: 30 days (recent historical trends)
• 5-minute aggregates: 90 days (dashboards and SLO tracking)
• 1-hour aggregates: 1 year (long-term patterns)

Prometheus doesn't downsample natively, but Thanos or Cortex both do this well. Here's a Thanos downsampling configuration:

retention:
  resolution_raw: 14d
  resolution_5m: 30d
  resolution_1h: 90d
downsample:
  period: 30d
  idle_timeout: 5m

This keeps raw data for 14 days, then automatically creates 5-minute and 1-hour aggregates for older data. Your storage shrinks by 80-90% after the first month while retaining enough detail for alerting and dashboards.

For logs, retention is equally important. Most teams store everything verbosely and pay dearly. Instead:

• Full structured logs: 7 days
• Sampled application logs: 30 days (keep 10% random sample)
• Error and warning logs: 90 days (always keep errors)

In practice, you'll rarely query logs older than a week anyway. Downsampling everything older than that week to errors-only saves 60-70% of log storage cost.

Sampling — Sample traces and verbose logs

Tracing is powerful but generates enormous volume. A single transaction might produce 20-50 spans. At 100 requests per second, that's 2,000-5,000 spans per second—unaffordable at 100% sampling for most organizations.

Adaptive sampling keeps you honest: sample based on what you need to know. High-error-rate requests get sampled at 100%. Slow requests get sampled heavily. Fast, successful requests get sampled at 1-5%.

A practical sampling strategy in code:

# Pseudocode sampling logic
def should_sample(request, response):
  # Always sample errors
  if response.status >= 500:
    return True
  
  # Always sample slow requests
  if response.duration_ms > 1000:
    return True
  
  # Sample 50% of slow-but-ok requests
  if response.duration_ms > 500:
    return random() < 0.5
  
  # Sample 5% of fast requests
  return random() < 0.05

This approach keeps your most valuable traces (errors and slow requests) at high fidelity while massively reducing volume for normal traffic. Most teams can maintain 99% visibility on problems while cutting trace volume by 95%.

For logs, sampling works similarly. Application debug logs are chatty and low-value in production. Sample them:

# Keep all errors and warnings
if level in ["ERROR", "WARN"]:
  log(entry)
# Sample info and debug
elif level == "INFO":
  if random() < 0.1:  # 10% sample
    log(entry)
elif level == "DEBUG":
  if random() < 0.01:  # 1% sample
    log(entry)

When you need full verbosity for debugging a specific issue, temporarily increase sampling via feature flags. Don't pay for full volume all the time.

Review — Delete unused dashboards and metrics

Dashboards accumulate like technical debt. Teams create them for one-off investigations, forget about them, and they sit there costing money. Similarly, many applications instrument metrics that nobody actually uses.

A monthly cost-reduction ritual:

1. Identify unmaintained dashboards. Export dashboard edit timestamps from Grafana. Anything untouched for 90 days is candidate for deletion.
2. Check dashboard usage. Grafana Cloud tracks which dashboards get viewed. Delete those with zero views in 30 days.
3. Find unused metrics. Query your monitoring system's API to list all metrics, then cross-reference with dashboard definitions and alert rules. Anything not in either is a cost drain.
4. Audit custom instrumentation. Every library and application sending metrics should be reviewed quarterly. Remove anything that doesn't drive operational decisions.

In one recent project, we discovered a "debug" metrics exporter that was running on all 200 production pods, sending 50 high-cardinality metrics each. That single oversight was costing $8,000 per month. It took 30 minutes to remove.

Create a simple checklist for new metrics:

• Does this metric answer an operational question?
• Is it used in a dashboard or alert rule?
• What cardinality does it have? (Run it in staging first.)
• How long do we need to retain this data?
• When will we revisit this decision?

Without this discipline, instrumentation sprawl will consume your budget.

Common mistakes

Mistake 1: Assuming all metrics cost the same. They don't. A metric with 100 series costs 100x less than one with 10,000. Cardinality is your primary cost driver, not the number of metrics. Audit the cardinality distribution first.

Mistake 2: Keeping raw metrics forever. Raw 15-second resolution data has diminishing value after a few weeks. Downsample aggressively. You'll still catch trends and alert on anomalies with 5-minute or 1-hour granularity, at a fraction of the cost.

Mistake 3: Sampling uniformly. Sampling 1% of all traces is crude. Bias sampling toward errors and slow requests where you actually need detail. Fast successful requests need far less sampling.

Mistake 4: Never reviewing what's being collected. Monitoring systems grow organically. Without quarterly reviews, you'll pay for months of unused dashboards and debug metrics. Build review into your operational rhythm.

Mistake 5: Treating cost optimization as a one-time project. Cardinality and instrumentation creep constantly. A metric safe at 100 series can balloon to 10,000 as your system grows. Make cost reduction a normal part of operational work, not an emergency project.

Key takeaways

• Cardinality is your biggest cost lever. Audit your top metrics by series count and fix unbounded labels (request IDs, user IDs, IPs).
• Downsample aggressively. Keep raw 15-second data for 1-2 weeks, then drop to 1-minute or 5-minute aggregates. Storage shrinks 80-90% with minimal visibility loss.
• Sample traces and logs adaptively. Capture 100% of errors and slow requests; sample fast successful traffic at 1-5%. You'll maintain visibility on problems while cutting volume by 95%.
• Review quarterly. Delete unused dashboards, audit custom instrumentation, and kill metrics that don't drive operational decisions.
• Cost optimization is continuous. Monitor your own monitoring costs like a metric and treat cost creep the same way you'd treat an alert storm.