Controlling cardinality: real-world lessons and examples

Controlling cardinality: real-world lessons and examples

Prometheus becomes slow and expensive when you're scraping thousands of unique metric combinations from a single endpoint. We've learned the hard way that unbounded labels—user IDs, request paths, order numbers—turn a lean monitoring stack into a resource hog that chews through memory and disk space. In this article, we'll walk through the practical tactics we use to keep Prometheus fast and affordable.

What is cardinality and why it matters

Cardinality is the number of distinct time series your monitoring system stores. A single metric like http_requests_total with labels method, endpoint, and status might generate dozens of series. If you label by user ID or order ID, you could end up with millions of series from a single metric—and Prometheus will struggle.

Each series consumes memory in the in-memory index, disk space in the TSDB, and CPU during queries. High cardinality scenarios cause:

• Memory bloat and out-of-memory crashes
• Slow query performance
• Higher storage costs
• Increased load on Grafana dashboards

The good news: cardinality is controllable once you know where to look.

Find offenders—topk by series count per metric

You can't fix what you don't see. Start by querying your Prometheus instance to identify which metrics are exploding in cardinality.

Connect to your Prometheus server and run this query in the web UI or via the HTTP API:

count by (__name__) ({__name__=~".+"})

This shows total series per metric. To see the top offenders:

topk(10, count by (__name__) ({__name__=~".+"}))

Now dig deeper. If http_requests_total has 50,000 series but you expected only a few hundred, inspect its labels:

count by (__name__, le) ({__name__="http_requests_total"})

We once discovered that a checkout service was instrumenting a metric with both the user_id and order_uuid labels. That innocent-looking cart endpoint was generating 200,000 unique series daily. By querying the metric metadata, we identified the culprit in minutes.

Use this PromQL to list all label names for a specific metric:

label_names({__name__="http_requests_total"})

Once you've found the offending metric and label, you have a choice: drop it at scrape time or fix the instrumentation. Most teams find a mix works best.

Drop labels—metric_relabel_configs remove noisy labels

When you can't change the application code immediately, metric_relabel_configs in your Prometheus scrape config acts as a safety valve. This lets you drop labels before they're stored in TSDB.

Example: your API service emits request_duration_seconds with labels method, endpoint, status, user_id, and trace_id. Both user_id and trace_id are cardinality bombs.

Add this to your prometheus.yml:

scrape_configs:
  - job_name: 'api-service'
    static_configs:
      - targets: ['localhost:8080']
    metric_relabel_configs:
      - source_labels: [__name__]
        regex: 'request_duration_seconds'
        action: 'keep'
      - source_labels: [__name__]
        regex: '.*'
        action: 'labelmap'
        replacement: ''
      - regex: 'user_id|trace_id'
        action: 'labeldrop'

Breaking this down:

• First rule keeps only the metric you want to process
• Second rule maps all labels (optional, clarifies intent)
• Third rule drops user_id and trace_id before storage

We recommend applying this approach as a temporary band-aid while you fix the root cause in your application code. Relabeling adds CPU load to Prometheus itself, so heavy use can slow down scraping.

Fix instrumentation—Never label with ids, emails or full URLs

The real fix lives in your application instrumentation. Engineers often label metrics with IDs thinking "this will help debugging"—but it destroys cardinality.

Here's a pattern we see repeatedly in NestJS services:

// Bad: cardinality explosion
const label_user_id = req.user.id; // 50k+ unique values
counter.inc({
  method: req.method,
  endpoint: req.route,
  user_id: label_user_id  // DON'T DO THIS
});

// Better: aggregate at query time if needed
counter.inc({
  method: req.method,
  endpoint: req.route,
  status: res.statusCode
});

The principle: labels should have low cardinality (< 100 unique values per label). Good label candidates are method (GET, POST, etc.), status_code (200, 404, 500), environment (prod, staging), and service_name. Bad candidates are anything that can take on thousands of values: user IDs, order IDs, session tokens, email addresses, full request paths, or UUIDs.

For detailed debugging, use structured logging or tracing instead. Your logs already have the full context; your metrics shouldn't.

In a trading platform we worked on, the team was labeling trade metrics with the customer account ID. This generated 100,000+ series from a single metric. After removing the account ID label and relying on logs for account-level breakdowns, the metric fell to under 500 series. Query performance improved 10x.

Set limits—sample_limit protects against explosions

Even with good practices, unexpected cardinality spikes happen. A misconfigured exporter, a logging loop, or a surge in unique values can flood your Prometheus. The sample_limit setting acts as a circuit breaker.

scrape_configs:
  - job_name: 'risky-service'
    static_configs:
      - targets: ['localhost:9090']
    sample_limit: 10000
    scrape_interval: 30s

If a scrape produces more than 10,000 samples, Prometheus will reject the entire scrape and log a warning. This prevents runaway cardinality from silently poisoning your database.

To calculate a reasonable limit, scrape the target a few times and count the samples. Most services generate 100–1000 samples per scrape. Set the limit 2–3x higher to account for legitimate growth, but low enough to catch cardinality bugs.

We also recommend series_limit at the global level to cap total series across all jobs:

global:
  scrape_interval: 15s
  # Reject if total series exceeds this
  series_limit: 5000000

This doesn't prevent problems—it surfaces them. Combined with alerting on scrape rejections, you'll catch cardinality leaks before they crash the server.

Common mistakes

Mistake 1: Assuming relabeling is a permanent fix. Relabeling is helpful short-term, but it doesn't solve the root problem. If your app is generating high-cardinality metrics, fix the instrumentation. Otherwise, you're burning CPU in Prometheus to hide a code smell.

Mistake 2: Labeling by full request path. A team scraped a web service and labeled every metric with the full URL path (/api/v1/users/123/orders/456). This created millions of series. Always aggregate: use /api/v1/users/{id}/orders/{id} or better yet, just label by the route handler name.

Mistake 3: Not monitoring cardinality growth. Set up a Grafana dashboard that charts count(count by (__name__) ({__name__=~".+"})) over time. When cardinality spikes, you'll see it immediately and can trace the cause.

Mistake 4: Setting sample_limit too high. If you set it to 1 million samples, it won't protect you. Base the limit on your expected usage with a small buffer—typically 5–10% headroom.

Key takeaways

• Find cardinality offenders using topk(10, count by (__name__) (...)) queries against your Prometheus server
• Drop noisy labels with metric_relabel_configs in prometheus.yml as a short-term fix
• Fix the root cause by removing high-cardinality labels from your application instrumentation—use logs or traces for detailed debugging instead
• Set sample_limit and series_limit to catch cardinality explosions early
• Monitor cardinality growth trends in Grafana so spikes don't surprise you in production