Grafana variables: Monitoring best practices explained

Grafana variables: Monitoring best practices explained. Reusable, filterable dashboards.

Opening Hook

On Day 20 of our 100-day Prometheus, Grafana & monitoring journey, we're tackling one of the most powerful yet underused features in Grafana: variables. When you're managing dozens of dashboards across multiple services, teams, and environments, the ability to build reusable, filterable dashboards becomes essential. Grafana variables let you parameterize your queries, annotations, and even panel titles, turning static dashboards into dynamic tools that adapt to your needs.

Why Grafana Variables Matter

Before we dive into the mechanics, let's establish why this matters. A well-designed dashboard with variables reduces duplication across your organization. Instead of maintaining separate dashboards for production, staging, and development environments, you create one dashboard that switches contexts via a dropdown. Your teammates don't need to clone and modify queries—they simply select the relevant scope and drill into the data they need.

Variables also improve query performance and reduce cognitive load. When you filter at the variable level, your Prometheus queries run faster because you're not fetching metrics for every instance or service. And when your dashboard adapts to the selected environment or service, viewers spend less time hunting for the right view and more time investigating problems.

Using label_values() to Populate Dropdowns

The foundation of dynamic dashboards is the label_values() function. This queries your Prometheus metrics directly and extracts unique label values, which then populate dropdown menus in Grafana.

Here's a practical scenario: you're monitoring a multi-tenant SaaS platform with metrics tagged by tenant_id. Rather than hardcoding tenant IDs into every query, you create a variable that automatically discovers all active tenants.

To set this up in Grafana, navigate to your dashboard settings and create a new variable with these settings:

Name: tenant_id
Type: Query
Data source: Prometheus
Query: label_values(http_request_duration_seconds, tenant_id)
Regex: (.*)

This queries any metric tagged with tenant_id, extracts the unique values, and displays them as options. When a user selects a tenant, Grafana stores that choice in $tenant_id, which you then reference in your Prometheus queries.

In your query, you'd write:

rate(http_request_duration_seconds{tenant_id="$tenant_id"}[5m])

If your tenant ID contains special characters, use the regex field to clean up the values. For instance, if label values are formatted as tenant-123-prod, you might extract just the numeric portion:

Regex: tenant-(\d+)-.*

This approach scales cleanly. When you onboard a new tenant and their metrics start flowing into Prometheus, the variable automatically includes them—no dashboard updates required.

Multi-Select and the All Option

Single-select variables are a starting point, but real-world monitoring often requires comparing across multiple dimensions simultaneously. Grafana's multi-select feature lets users pick several values at once, and the "All" option provides a convenient shortcut.

Consider a monitoring team overseeing a Kubernetes cluster with multiple namespaces. You want a variable that lets users select one namespace, several namespaces, or all of them.

Configure the variable like this:

Name: namespace
Type: Query
Data source: Prometheus
Query: label_values(container_cpu_usage_seconds_total, namespace)
Multi-select: On
Include All option: On
All value: .*

The "All value" field is crucial—it tells Grafana what regex pattern to use when "All" is selected. Using .* matches any namespace. In your Prometheus query, reference the variable with curly braces:

sum(rate(container_cpu_usage_seconds_total{namespace=~"$namespace"}[5m])) by (pod)

Notice the =~ operator, which performs regex matching. When a user selects multiple namespaces like production and staging, Grafana transforms $namespace into production|staging, and the regex match filters correctly. When they select "All," it becomes .*, matching every namespace.

This is far cleaner than forcing users to manually toggle between separate views. They stay in one dashboard and adjust scope with a single click.

Chained Variables and Cascading Filters

Variables become truly powerful when you chain them—making one variable's options depend on another variable's selection. This creates a natural hierarchy that mirrors your infrastructure.

Imagine a platform with multiple cloud regions, and within each region, multiple clusters. You want users to first select a region, then see only the clusters available in that region.

Set up two variables:

Variable 1: region

Name: region
Type: Query
Data source: Prometheus
Query: label_values(node_memory_MemAvailable_bytes, region)
Multi-select: Off

Variable 2: cluster

Name: cluster
Type: Query
Data source: Prometheus
Query: label_values(node_memory_MemAvailable_bytes{region="$region"}, cluster)
Multi-select: Off

The key is including region="$region" in the second query. Now, whenever a user changes the region dropdown, Grafana re-evaluates the cluster query filtered to that region. The cluster dropdown updates instantly to show only relevant options.

You can chain as many levels as needed, though three or four is typically the practical limit before the UX becomes cumbersome. Each level narrows the scope, reducing noise and helping users navigate complex infrastructures efficiently.

Using Variables in Queries and Panel Titles

Variables aren't limited to metric filters—they belong everywhere contextual information appears. Use them in panel titles, annotations, alert rules, and even dashboard descriptions.

For panel titles, wrap the variable reference in ${}:

CPU Usage for ${cluster} in ${region}

When a user selects us-west-2 as the region and web-prod as the cluster, the title dynamically updates to "CPU Usage for web-prod in us-west-2." This small detail improves clarity, especially when sharing dashboards or screenshots.

In Prometheus queries, you use $variable_name without braces:

avg(rate(http_requests_total{cluster="$cluster", region="$region"}[5m])) by (handler)

You can also use variables in threshold values or other numeric fields. For example, set a variable error_rate_threshold to 0.05, then reference it in an alert rule:

rate(http_requests_total{status="5xx"}[5m]) > $error_rate_threshold

This lets team leads adjust alert sensitivity from the dashboard without editing the underlying rules—useful for tuning thresholds during deployment windows or maintenance.

Common Mistakes

Forgetting to handle empty selections: If a variable has no "All" option and a user clears their selection, queries will fail. Always provide sensible defaults or an "All" option.

Over-chaining variables: Four or five cascading variables create a poor user experience. Keep hierarchies shallow and consider using a different dashboard layout if you need deeper filtering.

Mismatching regex patterns: If you use multi-select with =~ but forget to set an appropriate "All value," the regex won't match correctly. Test your variable with edge cases: single selection, multiple selections, and "All."

Hardcoding values alongside variables: It's tempting to mix hardcoded filters with variable filters, but this creates confusion. Be consistent—if you're filtering by namespace with a variable, don't also hardcode a region. Let variables handle all dynamic dimensions.

Not documenting variable purpose: In shared dashboards, document what each variable does. Add a note panel or update the dashboard description to explain the filtering logic, especially for chained variables.

Key Takeaways

• label_values() automatically discovers label values from your Prometheus metrics, eliminating the need to maintain hardcoded dropdown lists.
• Multi-select and "All" options make dashboards flexible enough for single-service investigations and cross-service comparisons in one view.
• Chained variables create cascading filters that mirror your infrastructure hierarchy, reducing cognitive load and helping teams navigate complex systems.
• Use variables throughout your dashboards—in titles, descriptions, queries, and alert thresholds—to keep context consistent and dynamic.
• Test variable behavior under all conditions: empty selection, single value, multiple values, and "All," to catch edge cases before sharing dashboards with your team.