Aggregation operators in PromQL: lessons from production

When your dashboards start drowning in thousands of metrics from microservices, load balancers, and databases, raw time-series data becomes noise rather than insight. The real power of Prometheus lies in aggregation operators in PromQL, which let us slice, dice, and summarize metrics to surface what actually matters. In this post, we'll explore the aggregation operators that have saved us countless debugging sessions in production.

Why aggregation matters in production systems

Early in our careers, we might query Prometheus naively—pulling every instance of a metric and staring at the raw results. This approach breaks down quickly. A single HTTP latency metric across 50 application instances generates 50 time series. Multiply that across databases, queues, caches, and sidecars, and you're looking at hundreds of thousands of series. Your brain can't parse that. Your dashboards become unreadable. Your alerting rules generate alert fatigue.

Aggregation operators solve this by reducing dimensionality while preserving information. They're not optional nice-to-haves—they're how we make Prometheus data actionable.

sum by (label) keeps chosen labels

The sum operator collapses multiple time series into fewer series by adding their values. The by clause tells Prometheus which labels to keep as grouping dimensions; all other labels are discarded.

Consider a typical scenario: you're monitoring HTTP request rate across an e-commerce platform. Your metric looks like this:

http_requests_total{instance="app-1",job="api",method="POST",path="/checkout",status="200"}
http_requests_total{instance="app-2",job="api",method="POST",path="/checkout",status="200"}
http_requests_total{instance="app-3",job="api",method="POST",path="/checkout",status="200"}
http_requests_total{instance="app-1",job="api",method="GET",path="/products",status="200"}
http_requests_total{instance="app-2",job="api",method="GET",path="/products",status="200"}

If you want to see total request rate per HTTP method (ignoring which instance handled it), you'd write:

sum by (method) (rate(http_requests_total[5m]))

This groups all series by the method label, adds them together, and gives you two time series: one for POST requests and one for GET requests. Prometheus internally discards the instance, path, and status labels during aggregation.

In production, we use this pattern constantly. You might want to know total traffic per service, or latency per database, or error rate per team's deployment. The key insight: sum by preserves exactly the labels you care about and nothing else.

A real example from a ride-hailing platform: we monitored request latency across 30 edge locations, each with multiple instances. Dashboards were too busy until we aggregated:

sum by (region) (rate(request_duration_seconds_sum[5m])) /
sum by (region) (rate(request_duration_seconds_count[5m]))

Suddenly we could see that our Europe region was degrading while Asia was fine. Without sum by, that pattern would've been invisible in the noise.

without (label) drops chosen labels

The without clause works inversely to by. Instead of specifying which labels to keep, you specify which labels to drop. Everything else becomes a grouping dimension.

This is useful when you know exactly what you want to ignore rather than what you want to keep. Suppose you have a metric with many labels, but you only want to drop ephemeral ones like pod_id or request_id:

sum without (pod_id, request_id) (metric_name)

This keeps all labels except those two. It's often clearer to read when you're aggregating high-dimensional metrics.

We hit this pattern during a CRM system redesign. Our database query metric had labels for schema, table, query_hash, instance, and connection_pool. We wanted to aggregate by table to see which tables were being hammered, but query_hash and instance varied constantly:

sum without (query_hash, instance) (
  rate(db_query_duration_seconds_sum[5m])
) /
sum without (query_hash, instance) (
  rate(db_query_duration_seconds_count[5m])
)

This told us that our users table was taking 800ms per query while orders table ran in 50ms—a clear signal to optimize.

One practical tip: avoid mixing by and without in the same query. It confuses readers and makes maintenance harder. Pick one style per query and stick with it.

topk for the noisiest series

When you're debugging a production incident, you often don't want an average or sum—you want the worst offenders. That's where topk shines. It returns the k largest values from a metric, ranked by current value.

Syntax: topk(5, metric_name) returns the top 5 time series by value.

Imagine you're investigating CPU usage spikes. Your system has hundreds of services, but you only care about the ones actually causing problems:

topk(10, rate(process_cpu_seconds_total[5m]))

This immediately shows you the 10 CPU-hungriest processes. No sorting through tables, no mental math. The result stays dynamic—as processes speed up or slow down, the ranking updates in real-time.

We've used topk extensively in a SaaS platform to find noisy customers. Each tenant's request rate was tracked, and during strange traffic patterns, we'd run:

topk(20, sum by (tenant_id) (rate(http_requests_total[5m])))

Often we'd find one tenant was hammering us with 10x their normal rate—helping us catch runaway client code before it tanked shared resources. Without topk, we'd have manually sorted a list of thousands of tenants.

Inverse ranking uses bottomk if you want the smallest values instead—useful for finding underutilized resources or anomalously low activity.

count for number of series

Sometimes you need to know not what the values sum to, but how many time series contributed to them. The count operator counts the number of non-empty series in a metric.

This sounds niche but proves invaluable for alerting on system topology changes. If you expect 50 application instances and count suddenly returns 30, you know something crashed.

count(up{job="api"})

This tells you how many API instances are up right now. If it drops below your expected minimum, trigger an alert:

count(up{job="api"}) < 40

In a travel platform we worked on, this pattern caught cascading failures early. We monitored:

count(up{job="booking_service"}) / count(up{job="booking_service"} offset 1h) < 0.8

If less than 80% of instances from an hour ago were still alive, we paged on-call immediately. It caught a deployment that rolled out bad code before it reached full saturation.

count also works with by and without. Counting by label is particularly useful:

count by (instance) (metric_name)

This shows how many unique label values contributed to an aggregation—essential for understanding cardinality and debugging label explosion issues.

Combining aggregation operators

Production queries rarely use a single operator in isolation. A realistic example from a trading platform:

topk(3, sum by (exchange) (
  rate(trade_orders_total{status="rejected"}[5m])
))

This finds the three exchanges with the most rejected orders in the past 5 minutes. The sum by aggregates across instances per exchange, then topk ranks the results. The mental model: first reduce dimensionality, then filter.

General principle: apply narrowing aggregations (sum by, avg by) before ranking operations (topk, bottomk). It's more efficient and easier to reason about.

Common mistakes

We've seen several patterns cause problems in production queries:

• Forgetting that sum loses labels: New users often assume sum(metric) keeps instance labels. It doesn't. You need sum by (instance) to preserve them. Read the docs carefully.
• Mixing instance and job in aggregations: The instance label identifies a specific scrape target; job groups targets. Aggregating by instance usually defeats the purpose. You almost always want to aggregate away instance and keep job.
• Using avg when you mean sum: Computing average latency across instances is correct for dashboards. But for alert thresholds, sum often makes more sense—you care about total load, not mean performance per instance.
• Aggregating gauge metrics with sum: Gauges like memory_bytes don't make sense to sum across instances. Use avg or max instead. Sum is for counters (requests_total, bytes_sent, etc.) that increase monotonically.
• topk without proper sorting context: topk ranks by current value, which changes second-to-second. For stable alerting, prefer explicit thresholds (e.g., rate(...) > 1000) over topk(10, ...).

Key takeaways

• Aggregation operators reduce dimensionality, turning thousands of time series into actionable data. Use them in nearly every production query.
• sum by (label) and sum without (label) collapse series while controlling which labels remain for grouping.
• topk(n, metric) surfaces the n largest values, perfect for finding worst offenders during incidents.
• count tracks series cardinality, essential for alerting on topology changes and catching cascading failures.
• Counters get sum; gauges get avg or max. Know your metric type.
• Apply narrowing aggregations before ranking operations for clarity and efficiency.
• Favoring explicit thresholds over dynamic ranking (topk) makes alerts more stable and predictable.