Labels and cardinality: a practical guide for developers

Introduction

If you've been running Prometheus for more than a few days, you've probably noticed performance issues that seemed to come out of nowhere. Your disk usage exploded, query times became glacial, and your monitoring system became the thing you needed to monitor. In nine out of ten cases, the culprit is cardinality—and it all comes down to how you use labels. Labels and cardinality are concepts that separate engineers who run Prometheus smoothly from those who are constantly fighting with it. This guide will show you exactly how to think about labels the right way, with concrete examples from production systems.

Understanding labels and metric dimensions

Labels in Prometheus are key-value pairs attached to metrics. They transform a single metric into multiple related metrics, creating what we call a time series. Without labels, you can only track one value per metric. With labels, you can break down that value by any dimension that matters.

For example, a simple metric without labels might look like this:

http_requests_total 15042

That tells you nothing useful. You have no idea where those requests went, what status codes came back, or which service handled them. Now add labels:

http_requests_total{service="api",method="GET",status="200"} 8492
http_requests_total{service="api",method="POST",status="201"} 2105
http_requests_total{service="api",method="GET",status="500"} 142
http_requests_total{service="web",method="GET",status="200"} 4303

Now you can answer real questions: How many 5xx errors did the API service return? Which service got the most traffic? Did POST request success rates change over time? Labels are what make metrics actionable.

The labels you choose should represent the dimensions that matter for understanding your system's behavior. Common examples include service name, HTTP method, status code, database name, cache hit/miss, or error type. These are stable, low-cardinality attributes that naturally partition your data.

Every label combination creates a new series

This is the critical insight that trips up most engineers. When you add a label to a metric, Prometheus doesn't just add one more value. It creates a separate time series for every unique combination of label values that appears in your data.

Imagine you're tracking response times for API calls with labels for service, endpoint, and status. If you have 10 services, 50 endpoints per service, and 5 possible status codes, that's 10 × 50 × 5 = 2,500 unique time series, all from a single metric.

This scales badly. Consider what happens if you add a label for request source (customer ID, user ID, or machine hostname):

• 10 services
• 50 endpoints
• 5 status codes
• 1,000 unique customer IDs

That's 2.5 million time series from one metric. And that's before you consider other metrics. Suddenly your Prometheus server is struggling, storage grows exponentially, and queries that used to complete in milliseconds timeout.

Here's a simplified example showing how this plays out in code. When you increment a counter with different label values:

// Pseudocode showing time series explosion
request_counter.WithLabelValues("api", "POST", "/users", 201).Inc()
request_counter.WithLabelValues("api", "POST", "/users", 400).Inc()
request_counter.WithLabelValues("web", "GET", "/home", 200).Inc()
request_counter.WithLabelValues("web", "GET", "/profile", 200).Inc()
// Each unique combination is stored separately

Each of these calls creates or updates a different time series. If you're running in a microservices environment with hundreds of instances, and you add labels for instance hostname or pod name, you've multiplied the problem by the number of instances running.

Never use user IDs or URLs with IDs as labels

This is the mistake that causes the most damage in production. The temptation is always there: I need to know which user caused a problem, so I'll add a user_id label. Seems reasonable. It's a catastrophe.

User IDs, request IDs, email addresses, or any naturally high-cardinality identifier will create millions of unique series instantly. If your user base grows, so does your cardinality, and there's no upper bound. This is how monitoring systems get taken down by their own metrics.

Similarly, never use URLs or paths as labels if they contain IDs. This query endpoint:

GET /api/users/12345
GET /api/users/67890
GET /api/users/11111

Should never produce three different label values. If you track the path as a label, you'll have one series per unique user accessed. On a moderately active system, that's thousands of new series per minute.

The right approach is to extract stable parts of the path:

http_requests_total{service="api",method="GET",endpoint="/users/{id}",status="200"}

Notice the /users/{id} label value. The placeholder stays constant regardless of which user was requested. Your query logic (middleware, instrumentation library, or custom code) should normalize these URLs before recording metrics.

If you truly need to investigate a specific user's impact, store that in logs or traces, not in metrics. Logs are built for high-cardinality data. Metrics are not.

High cardinality kills performance

The performance impact of unbounded cardinality is severe and multifaceted. Let's walk through what happens as cardinality grows.

Memory usage

Prometheus keeps active time series in memory. With millions of series, you need gigabytes of RAM just to track label combinations. A Prometheus instance that should fit in 2GB might need 16GB or more when cardinality is out of control.

Disk I/O and storage

Every series gets persisted. More series means more data written to disk, faster disk growth, and longer compaction times. A metric with unbounded cardinality can fill a 100GB disk in days on a moderately trafficked system.

Query performance

When you query a metric, Prometheus has to scan all matching series and aggregate results. With millions of series, even simple queries become slow. A query like rate(requests_total[5m]) might take seconds instead of milliseconds if it matches a million series.

Scrape performance

Scraping and parsing metrics takes longer with more series. Your Prometheus instance's CPU usage climbs, and scrapes might take longer than your scrape interval, causing cycles to be skipped.

We've seen production systems where adding a single poorly-chosen label (customer ID, for example) caused the monitoring system to become so slow that alerts didn't fire for hours. The irony is bitter: better instrumentation broke the entire monitoring setup.

Identifying high-cardinality problems

Prometheus provides tools to find cardinality problems before they become critical. Use the topk function with count by to see which metrics have the most series:

topk(10, count by(__name__) (count by(__name__, __value__) (ALERTS) ignoring(__value__)))

Or use the Prometheus API to get cardinality information:

curl http://localhost:9090/api/v1/label/__name__/values | head

Many teams also run cardinality monitoring tools like prometheus-cardinality-exporter to get alerts when metrics exceed expected cardinality ranges.

Check your scrape targets tab in Prometheus UI. Look for targets that expose way more series than expected. If a service that should expose 100 metrics shows 100,000, investigate immediately.

Common mistakes

Mistake 1: Treating labels like dimensions in a data warehouse. In a data warehouse, adding dimensions is free. In Prometheus, each new dimension multiplies your cardinality. Resist the urge to add labels just because you think you might query them someday.

Mistake 2: Adding labels at request time without normalization. If your instrumentation code does this:

counter.WithLabelValues(request.URL.Path, request.UserID).Inc()

You're creating a new series for every unique URL and user combination. Your monitoring system will be dead within hours on any real traffic. Always normalize high-cardinality values before they become labels.

Mistake 3: Forgetting about instance labels. Some instrumentation libraries automatically add instance or pod name labels. On a system with 100 instances, this multiplies your cardinality by 100. Check your instrumentation library's documentation and disable automatic high-cardinality labels if needed.

Mistake 4: Not planning labels upfront. Adding labels is easy. Removing them later is hard—you have to deal with alerting rules, dashboard queries, and retention policies all breaking. Think about your label schema before deploying to production.

Key takeaways

• Labels add dimensions to metrics, turning a single value into multiple related time series.
• Every unique combination of label values creates a new series; cardinality is the count of all unique series for a metric.
• Never use unbounded identifiers (user IDs, request IDs, email addresses) as label values.
• Never use URLs or paths containing IDs as labels; normalize them to patterns like /users/{id} first.
• High cardinality kills Prometheus performance across memory, disk, query speed, and scrape latency.
• Plan your label schema upfront and keep it stable; changing labels in production is painful.
• Use Prometheus querying tools to monitor your own cardinality and set up alerts for unexpected growth.
• If you need to track high-cardinality data, use logs or traces instead of metrics.