Vector matching in PromQL: practical tips and trade-offs

Vector matching in PromQL: practical tips and trade-offs

When you combine metrics in PromQL queries, you're not just doing arithmetic—you're joining data across different label dimensions. Get the join logic wrong, and your dashboards fill with gaps or misleading aggregations. Understanding vector matching in PromQL and how to control which labels participate in that join is critical for writing reliable alert rules and dashboards.

Why vector matching matters

In production systems, metrics often carry labels that identify instances, services, or regions. When you query a single metric, PromQL returns results grouped by all label combinations. But when you combine two metrics in an operation—say, dividing request count by request duration—Prometheus needs to know which label values should be paired together. This is vector matching, and it's where many queries go wrong.

Consider a real scenario: you're computing request latency percentiles by dividing a sum of request durations by a count of requests. Both metrics have labels for service, method, and instance. If you don't tell Prometheus how to match them, it tries a default join strategy that might silently drop data or produce unexpected results. Knowing the join rules lets you write predictable queries and avoid surprises in alerts.

One-to-one matching by default

By default, PromQL uses one-to-one matching. This means each vector element on the left side of an operator is paired with exactly one element on the right side, matching on all common label names.

Here's what that looks like in practice. Suppose you have:

• http_requests_total{service="api", method="GET", instance="i-001"} = 1000
• http_request_duration_seconds{service="api", method="GET", instance="i-001"} = 50

When you write:

http_request_duration_seconds / http_requests_total

Prometheus matches on the common labels (service, method, instance). If both metrics have the same label values, the division succeeds. If one metric is missing a label combination that exists in the other, Prometheus drops that combination from the result.

This default behavior is often what you want. It ensures type safety: you don't accidentally divide requests from service A by durations from service B. But it also means incomplete label alignment causes silent data loss. Always verify that both sides of your binary operation have matching label sets.

Controlling labels with on() and ignoring()

Sometimes you don't want to match on all common labels. Use the on() and ignoring() modifiers to refine the join criteria.

on() specifies exactly which labels to use for matching. For example:

rate(http_requests_total[5m]) / on(service, method) rate(http_request_duration_seconds[5m])

This pairs metrics only by service and method, ignoring differences in instance, endpoint, or other labels. If the left side has multiple instances and the right side has a single aggregated value, they'll now match.

ignoring() is the inverse: it tells Prometheus to match on all labels except the ones you list. This is useful when you know a specific label shouldn't participate in the join:

instance_memory_usage / ignoring(mode) instance_memory_limit

Here, both metrics might have a mode label (e.g., "heap" or "stack"), but you want to match by instance only, allowing multiple mode values on each side.

A practical scenario: you're building a dashboard for a multi-region system. Your request count is aggregated by region, but your error count is also aggregated by region. You want to compute error rate by region:

sum by(region) (rate(http_requests_total[5m])) / on(region) sum by(region) (rate(http_errors_total[5m]))

Without the on(region) modifier, Prometheus would try to match on all labels, potentially excluding regions where only one metric has data. The explicit on() ensures you get a result for every region that appears in both metrics.

group_left for many-to-one joins

One-to-one matching isn't enough for every scenario. Sometimes you need to pair multiple elements on one side with a single element on the other. This is a many-to-one join, and it requires the group_left modifier (or group_right).

Real example: you have per-instance CPU metrics, but you want to divide them by a per-service SLA. The service SLA is defined once per service, while CPU metrics exist for every instance of that service:

rate(node_cpu_seconds_total[5m]) / on(service) group_left(instance) node_service_sla

The group_left(instance) tells Prometheus: "I'm on the left side, and I have extra labels (instance) that won't participate in the match. For each service, pair my left-side elements (which have service and instance) with the single right-side element (which has only service)." The result preserves the instance label from the left.

Without group_left, Prometheus would apply one-to-one matching and likely return no results, because the left side has more label dimensions than the right side.

Another example: you're computing utilization percentages by pairing actual usage metrics with limit metadata:

container_memory_usage_bytes / on(pod_name) group_left() pod_memory_limit_bytes

The pod_memory_limit_bytes` might have extra labels like namespace and pod_name, while usage has the same labels. The group_left() with empty parentheses means: "I'm preserving all non-matching labels from the left side." If you had extra labels on the right, you'd list them: group_left(extra_right_label).

Note: be cautious with many-to-one joins in alert rules. If the right side matches multiple elements unexpectedly, you could trigger fires alerts multiple times or lose alerts entirely. Always test the join cardinality with a separate query first.

Adding metadata labels from info metrics

A common pattern in production is storing static metadata—version, owner, environment—in separate info metrics that have value 1. You then use many-to-one joins to add these labels to your operational metrics.

For instance, a telemetry system might export:

node_info{hostname="prod-web-01", environment="production", owner_team="platform"} 1

Your CPU and memory metrics only have hostname. To attach environment and owner information, you join:

rate(node_cpu_seconds_total[5m]) / on(hostname) group_left(environment, owner_team) node_info

The result now has environment and owner_team, which you can use for routing alerts or segmenting dashboard panels. The group_left(environment, owner_team) explicitly lists the extra labels from the right side that you want to keep.

This pattern is powerful but carries a caveat: the info metric must have a 1:1 relationship with the join key (e.g., one info entry per hostname). If you accidentally have multiple info entries for the same hostname, the many-to-one join will produce multiple results, silently duplicating data. Always validate your info metric cardinality.

A safer approach in some cases is to use Prometheus recording rules to pre-join metadata, reducing the risk of mismatches at query time:

- record: node:cpu:rate5m
  expr: rate(node_cpu_seconds_total[5m]) / on(hostname) group_left(environment) node_info

Then query node:cpu:rate5m directly. The join happens once during rule evaluation, and your alerts and dashboards consume clean, consistent data.

Common mistakes

• Forgetting about default one-to-one matching: New users often expect all common labels to participate in joins and are surprised when data disappears. Always check label cardinality before and after your operation. Use group() to see what labels each metric has: group(metric1) and on() group(metric2).
• Misusing group_left on cardinality mismatches: If both sides have one-to-one cardinality on the join key but you use group_left, you'll get duplicate results. Use group_left only when the left side truly has higher cardinality (more granular labels).
• Not testing joins in alerts: A query that looks correct in Grafana might produce unexpected results when evaluated for an alert. Always manually fire your alert condition and verify it triggers on the expected instances and timing.
• Mixing on() and group_left carelessly: Forgetting to list all extra labels in group_left() means those labels disappear from the result, potentially breaking downstream aggregations or losing important routing information.
• Assuming info metric 1:1 cardinality: If an info metric scrapes from multiple endpoints or has stale entries, you might accidentally create many-to-one joins where you expected one-to-one. Monitor cardinality with count by(join_key)(metric) and verify it's 1.

Key takeaways

• One-to-one matching is PromQL's default: both sides must have identical label values on all common labels, or the result is dropped.
• Use on() to explicitly specify which labels participate in the join; use ignoring() to exclude certain labels from matching.
• group_left (or group_right) enables many-to-one joins when one side has finer-grained labels than the other; always verify cardinality first.
• Info metrics are a powerful way to attach static metadata, but validate their 1:1 relationship with the join key to avoid silent duplication.
• Test joins in a separate query before embedding them in alerts; cardinality mismatches often show up only under certain load or scrape timing.
• Recording rules can pre-join metadata, reducing complexity and risk in alert and dashboard queries.