Relabeling in Prometheus: a hands-on guide for engineers

Prometheus gives you raw power to ingest and store metrics, but without deliberate labeling strategy, your dashboards and alerts become noise. Relabeling in Prometheus: a hands-on guide for engineers covers the two mechanisms that let you shape targets and metrics before they land in your time-series database—and we'll show you exactly how to use them in production.

Why relabeling matters in your monitoring stack

Every metric that enters Prometheus carries labels. Those labels determine how you group data in Grafana, how you route alerts in Alertmanager, and how efficiently you query. Without intentional relabeling, you inherit whatever labels your exporters and service discovery provide—which is often messy. You might get instance names instead of pod names, redundant environment tags, or metrics you don't need at all.

Relabeling happens at two points in the pipeline:

• Target relabeling shapes the labels on scrape targets before Prometheus connects to them.
• Metric relabeling modifies labels on individual samples after scraping, or drops samples entirely.

Both use regex patterns and a set of action verbs. Learning to wield them cleanly saves disk space, improves query performance, and keeps your alerts from drowning in cardinality.

relabel_configs for targets

Target relabeling runs on every target discovered by your service discovery mechanism—Consul, Kubernetes, file, static configs, whatever. It modifies the labels that describe the scrape endpoint itself, not the metrics it produces.

Consider a Kubernetes cluster where the Prometheus Operator auto-discovers pods. Each pod has metadata labels like pod.name, pod.namespace, and pod.ip. But you want human-readable labels in your metrics: service, env, and team. That's where target relabeling steps in.

global:
  scrape_interval: 15s

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Keep only pods in 'prod' and 'staging' namespaces
      - source_labels: [__meta_kubernetes_namespace]
        regex: 'prod|staging'
        action: keep

      # Rename pod name to 'pod_name'
      - source_labels: [__meta_kubernetes_pod_name]
        target_label: pod_name
        action: replace

      # Extract service name from pod label
      - source_labels: [__meta_kubernetes_pod_label_app]
        target_label: service
        action: replace

      # Set environment based on namespace
      - source_labels: [__meta_kubernetes_namespace]
        regex: 'prod'
        target_label: env
        replacement: 'production'
        action: replace
      - source_labels: [__meta_kubernetes_namespace]
        regex: 'staging'
        target_label: env
        replacement: 'staging'
        action: replace

      # Use pod IP and port for scrape address
      - source_labels: [__meta_kubernetes_pod_ip, __meta_kubernetes_pod_container_port_number]
        separator: ':'
        target_label: __address__
        action: replace

Let's unpack this step by step:

• The action: keep on the namespace acts as a filter—only pods in prod or staging are scraped. Targets that don't match are dropped before any metric is collected.
• action: replace copies or transforms a label. source_labels tells Prometheus which metadata label to read; target_label is the output label on scraped metrics.
• When you have multiple source labels, separator joins them (here, colon-separated IP:port for the scrape address).
• The special label __address__ controls where Prometheus connects. By default it's pod_ip:8080, but you can override it if your exporter listens on a different port or hostname.

metric_relabel_configs for samples

After Prometheus scrapes metrics from a target, metric relabeling modifies or drops individual time-series samples. This happens after scraping, so you're working with real metric data, not discovery metadata.

Metric relabeling is useful when an exporter produces dozens of metrics but you only care about a handful. Dropping unwanted metrics early saves storage and query overhead.

scrape_configs:
  - job_name: 'node-exporter'
    static_configs:
      - targets: ['localhost:9100']
    metric_relabel_configs:
      # Drop memory-related metrics we don't need
      - source_labels: [__name__]
        regex: 'node_memory_Slab.*'
        action: drop

      # Drop high-cardinality device metrics
      - source_labels: [__name__]
        regex: 'node_network_.*'
        action: drop

      # Keep only specific CPU metrics
      - source_labels: [__name__]
        regex: 'node_cpu_seconds_total'
        action: keep

      # Rename instance label to hostname
      - source_labels: [instance]
        target_label: hostname
        action: replace

      # Add a 'cluster' label to all samples
      - target_label: cluster
        replacement: 'us-west-2'
        action: replace

Key differences from target relabeling:

• Metric relabeling works on the actual metric name (__name__) and labels from the scraped data, not discovery metadata.
• action: keep with a regex on __name__ is a whitelist—only metrics matching the pattern are kept. Everything else is dropped.
• action: drop is the inverse: metrics matching the pattern are discarded.
• You can add arbitrary labels with action: replace and no source_labels—the replacement value becomes the label.

This approach works well for large exporters like node_exporter that emit 100+ metrics. You might only graph disk, network, and CPU; metric relabeling lets you prune the rest before they hit disk.

Keep or drop by regex

The regex patterns in relabeling are anchored. That means ^pattern$ is implicit—your pattern must match the entire label value.

metric_relabel_configs:
  # This keeps ONLY node_cpu_seconds_total
  - source_labels: [__name__]
    regex: 'node_cpu_seconds_total'
    action: keep

  # This drops metrics starting with 'node_memory'
  - source_labels: [__name__]
    regex: 'node_memory.*'
    action: drop

  # Multiple patterns: keep metrics with 'http' OR 'grpc' in the name
  - source_labels: [__name__]
    regex: '.*(http|grpc).*'
    action: keep

  # Drop any metric with a 'le' label (helpful to drop histogram buckets)
  - source_labels: [le]
    regex: '.+'
    action: drop

A practical scenario: your application's Prometheus client library exports many internal metrics you don't care about. Use a keep rule to whitelist only your application metrics. This cuts cardinality, speeds up queries, and focuses your dashboards on what matters.

  # Keep only app_* and http_* metrics
  - source_labels: [__name__]
    regex: '(app|http)_.*'
    action: keep

Add environment labels

In a multi-tenant or multi-environment setup, you often want every metric to carry an env, region, or tenant label. Rather than relying on exporter configuration, you can inject these at the Prometheus scrape level.

global:
  scrape_interval: 15s
  external_labels:
    env: production
    region: us-west-2

scrape_configs:
  - job_name: 'api-servers'
    consul_sd_configs:
      - server: 'consul.service.consul:8500'
        datacenter: 'us-west-2'
    relabel_configs:
      # Use the consul service tag as 'tier'
      - source_labels: [__meta_consul_service_tags]
        regex: '.*tier=([^,]+).*'
        target_label: tier
        replacement: '${1}'
        action: replace

      # Use the datacenter from service discovery
      - source_labels: [__meta_consul_datacenter]
        target_label: dc
        action: replace

      # Add a static 'owner' label for all API servers
      - target_label: owner
        replacement: 'platform-team'
        action: replace

Global external labels (top-level external_labels) are added to every metric scraped by this Prometheus instance. They're useful for distinguishing metrics from different Prometheus servers in a federation setup.

Job-level labels from relabeling are more flexible. You can use regex captures (the ${1} syntax) to extract parts of a label and reuse them. In the example above, we pull the tier value from a comma-separated tag list and promote it to its own label.

Common mistakes

Forgetting the anchors in regex. If you write regex: 'http', it won't match my_http_requests_total the way you expect in keep/drop rules. The regex engine in Prometheus anchors patterns, so you need regex: '.*http.*' to match that substring.

Using relabeling to hide metrics instead of dropping them. Renaming a metric or dropping a label doesn't save storage—the data was still scraped. Use metric relabeling with action: drop on __name__ to avoid scraping unwanted metrics in the first place.

Modifying __address__ incorrectly. The address must be a valid host:port. If you break it, Prometheus will fail to connect to the target. Test your regex capture groups carefully with promtool check config.

Over-relabeling in target configs. Target relabeling runs on every target during discovery refresh, which can be frequent. Complex regex chains slow down reconciliation. Keep it lean.

Confusing target and metric relabeling. Remember: target relabeling happens before scraping (metadata only). Metric relabeling happens after scraping (on actual samples). If you're trying to filter by metric name, use metric relabeling. If you're filtering by pod namespace, use target relabeling.

Key takeaways

• Target relabeling shapes metadata from service discovery; metric relabeling modifies scraped time-series. Use each at the right stage in the pipeline.
• Whitelist metrics with action: keep on __name__ to reduce cardinality and storage cost early.
• Regex patterns are anchored; use .*pattern.* to match substrings in keep/drop rules.
• Inject environment, region, and team labels via relabeling rather than hardcoding them in every exporter config.
• Test relabeling rules with promtool check config and validate them in a staging environment before production rollout.
• Keep relabeling rules readable and maintainable; complex chains become hard to debug when targets stop appearing.