Recording rules: what every Monitoring developer should know

The Cost of On-Demand Metric Queries

You're staring at a dashboard during a production incident. A query that should return in milliseconds is timing out. Your alert that's supposed to fire within seconds is stuck evaluating a complex expression across millions of data points. This is where recording rules: what every Monitoring developer should know becomes essential knowledge.

Recording rules solve a fundamental problem: expensive queries become cheaper when their results are precomputed and stored. Instead of calculating the same metric combination repeatedly, Prometheus evaluates the rule on a schedule and persists the output as a new time series. For teams running production systems at scale, this distinction between on-demand and precomputed metrics is the difference between snappy dashboards and timeouts, between reliable alerts and missed incidents.

Store Results as New Series

A recording rule is a Prometheus configuration that tells the server: "Evaluate this expression every N seconds and write the result as a new metric." The output becomes a real time series, queryable just like any other metric, but with guaranteed freshness based on your evaluation interval.

Consider a typical scenario. You have a microservice that exposes request latency percentiles at collection time. But you also want to track request_latency_95p_by_service across your entire infrastructure, combining data from multiple instances. Without recording rules, every dashboard panel and alert rule that needs this metric must execute the aggregation query independently. At query time, Prometheus scans millions of raw samples, performs the percentile calculation, and returns the result.

With a recording rule, you define the expression once:

groups:
  - name: latency_rules
    interval: 30s
    rules:
      - record: latency:request_duration_seconds:p95:5m
        expr: histogram_quantile(0.95, rate(request_duration_seconds_bucket[5m]))

Now Prometheus evaluates this expression every 30 seconds and stores the result as a new series called latency:request_duration_seconds:p95:5m. Your dashboards query this precomputed metric instead of running the histogram_quantile calculation on millions of data points. The metric is written to the TSDB like any ingested metric, persisted to disk, and queryable by any client.

The benefit compounds with complexity. If you have ten dashboard panels each using variations of this query, you've reduced the evaluation cost from ten expensive calculations to one. If an alert rule references this metric, it evaluates instantly. If you're running queries across a federation of Prometheus instances, the difference between precomputed and on-demand becomes the difference between sub-second response and query rejection due to resource exhaustion.

Faster Dashboards and Alerts

Dashboard performance directly impacts incident response. When a production incident is unfolding, you need to see system state immediately. A 10-second dashboard load time costs you real investigation time.

Suppose you're building a service health dashboard for a platform serving millions of requests daily. One panel shows "error rate by service." The naive query:

sum by (service) (rate(http_requests_total{status=~"5.."}[5m])) 
/ 
sum by (service) (rate(http_requests_total[5m]))

This query scans request metrics across all services and instances, calculates per-second error rates over a five-minute window, aggregates by service, and performs division. On a busy system with thousands of time series, this takes 2–5 seconds per execution. Multiply that across eight dashboard panels and you're waiting 20+ seconds for the dashboard to load.

With recording rules evaluated every 30 seconds, the same metric returns in milliseconds. Your dashboard becomes responsive. Teams investigating incidents spend time on the problem, not on Grafana load screens.

Alerts see similar gains. An alert rule evaluating expensive queries is evaluated repeatedly on the alert rule interval (typically 30–60 seconds). If the query takes 10 seconds to evaluate, you're consuming significant CPU on every cycle. More importantly, evaluation latency affects alert delay. If the rule takes 8 seconds to execute and your interval is 30 seconds, a true alert condition might not fire until 38+ seconds have elapsed. By the time the alert reaches your on-call team, you're already several seconds into the incident.

Recording rules evaluate once per interval and persist instantly. Alerts referencing precomputed metrics fire with predictable, low latency. You're alerting on fresh data without the computational cost.

Name with level:metric:operations

Prometheus recommends a specific naming convention for recording rules to make their purpose immediately clear: level:metric_name:operation. This pattern groups related metrics, indicates aggregation scope, and documents the transformation applied.

Breaking this down:

• level: The aggregation scope. Common values are instance (per-machine), job (per-application), service (business-level), or custom dimensions like region or shard.
• metric_name: The base metric being transformed. Keep it close to the original exported metric name.
• operation: What's being calculated. Examples: rate5m, p95, total, per_second, ratio.

Real examples from production systems:

groups:
  - name: aggregate_rules
    interval: 30s
    rules:
      # Per-instance query latency, 99th percentile
      - record: instance:http_request_duration_seconds:p99:5m
        expr: histogram_quantile(0.99, rate(http_request_duration_seconds_bucket[5m]))

      # Per-service error rate
      - record: service:http_requests:error_ratio:5m
        expr: |
          sum by (service) (rate(http_requests_total{status=~"5.."}[5m]))
          /
          sum by (service) (rate(http_requests_total[5m]))

      # Per-job CPU usage
      - record: job:node_cpu:usage_percent:1m
        expr: 100 - (avg by (job) (rate(node_cpu_seconds_total{mode="idle"}[1m])) * 100)

      # Regional database connection pool utilization
      - record: region:db_connections:utilization:10m
        expr: |
          sum by (region) (mysql_global_status_threads_connected)
          /
          sum by (region) (mysql_global_variables_max_connections)

This naming convention makes it obvious what the metric represents without reading the query. When you see service:http_requests:error_ratio:5m in a dashboard or alert, you immediately know it's a per-service error ratio calculated over five minutes. Operators and on-call engineers can understand your metrics at a glance.

The convention also scales. As your monitoring system grows to hundreds of rules, consistent naming keeps metrics organized and discoverable. When you're searching Grafana for latency metrics, prefixing with the level or operation makes finding the right series trivial.

Evaluate on the Rule Interval

Recording rules are evaluated on a fixed schedule. The interval field controls how often Prometheus executes the rule and writes the result. Getting this right is critical for both accuracy and system load.

The interval you choose depends on your use case:

• 30 seconds: Default for most rules. Balances freshness with computational cost. Suitable for alerts and dashboards where sub-minute granularity is sufficient.
• 1–5 minutes: For heavyweight aggregations or long-term trend analysis. Example: calculating 95th percentile CPU across all services in a region.
• 10 seconds or less: Only when latency is critical and the query is extremely cheap. Rare in practice.

Consider the freshness-cost tradeoff. A rule that evaluates every 10 seconds executes 8,640 times per day. A rule that evaluates every 5 minutes executes only 288 times. If each evaluation takes 500ms of CPU time, the difference is significant on a system with hundreds of rules.

Here's a practical example:

groups:
  - name: fast_rules
    interval: 30s
    rules:
      # Alert-critical: fire latency threshold breaches quickly
      - record: service:request_latency:p99:5m
        expr: histogram_quantile(0.99, sum by (le, service) (rate(request_duration_seconds_bucket[5m])))

  - name: slow_rules
    interval: 5m
    rules:
      # Reporting metric: daily capacity trending
      - record: service:peak_requests_per_second:daily
        expr: max_over_time(sum by (service) (rate(http_requests_total[1m]))[1d:1m])

The first rule fires every 30 seconds because alerts depend on it and need fast response. The second rule fires every five minutes because it's used for capacity planning and daily trending; there's no need to recalculate every 30 seconds.

Be aware: the interval also determines the resolution of the recorded metric. If you record every five minutes, your resulting series has samples only at five-minute boundaries. You can't query sub-minute granularity from that series. Plan accordingly.

Common Mistakes

Recording rules that are too granular. Teams sometimes record one rule for every possible aggregation. This becomes a maintenance nightmare. Instead, record the queries that are actually expensive and used repeatedly. Your base metrics are still queryable; recording rules should handle aggregations.

Interval misalignment with alert intervals. If your alert rule evaluates every 30 seconds but references a metric recorded every five minutes, your alert fires only every five minutes at best. This delays incident detection. Ensure recording intervals are equal to or shorter than dependent alert intervals.

Forgetting to update dependent queries when the rule changes. If you modify a recording rule's expression, all dashboards and alerts using that metric will now see different data. Document the rule's purpose and any breaking changes. Consider versioning by creating a new metric name instead of modifying existing rules.

Recording high-cardinality metrics. If your expression includes labels with very high cardinality (thousands of unique values), the recorded series multiplies storage and query cost rather than reducing it. Example: recording per-request-id metrics instead of aggregating by service. Cardinality kills recording rules.

Setting intervals too short. Recording every 10 seconds when 30 or 60 seconds is fine wastes CPU and storage. Start with 30-second intervals and increase only if you have a specific latency requirement.

Key Takeaways

• Recording rules precompute expensive expressions and store results as queryable time series, dramatically reducing dashboard and alert evaluation latency.
• Use the level:metric:operation naming convention to make rule purpose self-documenting and improve metric discoverability across your team.
• Choose intervals based on freshness requirements and computational cost; 30 seconds is the typical default, but alert-critical rules might need shorter intervals.
• Record only the queries that are genuinely expensive and reused; base metrics are always queryable on-demand.
• Avoid high-cardinality recording rules; they amplify storage and query cost instead of reducing it.
• Document rule changes and consider creating new metrics instead of modifying existing rules to avoid breaking dependent alerts and dashboards.
• Record one level of aggregation per rule for clarity; chain rules if needed rather than trying to do everything in one expression.