Service discovery: Monitoring best practices explained

Introduction

Manually typing every server IP and port into your Prometheus configuration becomes a nightmare as soon as you move beyond a handful of machines. Service discovery: monitoring best practices explained is about automating the process of finding, registering, and monitoring your targets so your observability stack scales with your infrastructure. Whether you're running in Kubernetes, across EC2 instances, or using Consul, the right discovery strategy eliminates configuration drift, reduces human error, and lets your team focus on what matters: understanding system behavior.

Why service discovery matters in production

Hard-coded target lists couple your monitoring system to your infrastructure in ways that break down quickly. When a new service spins up or an instance fails over, your scrape configs are stale. You either miss metrics entirely or spend time manually updating configuration files and reloading Prometheus.

Service discovery decouples these concerns. Prometheus queries a source of truth—Kubernetes, EC2, Consul, or even a simple file—to dynamically build its list of targets. When infrastructure changes, Prometheus automatically adjusts. This isn't just convenience; it's the difference between reliable observability and a monitoring system that lags behind reality.

Kubernetes service discovery

If you're running containers on Kubernetes, Prometheus has first-class support via the Kubernetes SD config. This integration watches the Kubernetes API and automatically discovers pods and services across namespaces.

Here's how you configure it:

scrape_configs:
  - job_name: 'kubernetes-pods'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: 'true'
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__
      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: kubernetes_namespace
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: kubernetes_pod_name

This config watches all pods. The relabel rules filter to only those annotated with prometheus.io/scrape: "true", extract the metrics path and port from pod annotations, and attach namespace and pod name as labels. When a deployment scales from 3 to 10 replicas, Prometheus discovers the new pods automatically within seconds.

For a real-world example: your API team deploys a new payment service. It includes the right annotations in its pod spec. Prometheus picks it up immediately without any intervention from your platform team. Dashboards start populating with metrics from the new service instantly.

EC2 service discovery

In AWS environments using EC2 instances, Prometheus can query the EC2 API directly. This works well for traditional infrastructure or hybrid setups where containers haven't fully taken over.

Configure it like this:

scrape_configs:
  - job_name: 'ec2'
    ec2_sd_configs:
      - region: us-east-1
        port: 9100
        access_key: YOUR_ACCESS_KEY
        secret_key: YOUR_SECRET_KEY
    relabel_configs:
      - source_labels: [__meta_ec2_tag_Environment]
        action: replace
        target_label: environment
      - source_labels: [__meta_ec2_tag_Service]
        action: replace
        target_label: service
      - source_labels: [__meta_ec2_instance_id]
        action: replace
        target_label: instance_id
      - source_labels: [__meta_ec2_private_ip]
        action: replace
        target_label: __address__
        regex: (.+)
        replacement: ${1}:9100

Prometheus queries the EC2 API every 60 seconds (configurable) and scrapes all instances, pulling in tags as labels. When your infrastructure team launches a new web server with tags Environment: production and Service: web, Prometheus finds it and starts collecting metrics within the refresh interval.

Use IAM roles and policies rather than hardcoding credentials. This keeps secrets out of version control and makes credential rotation simpler.

Consul service discovery

If you're using Consul for service registration, Prometheus integrates cleanly. Services register themselves in Consul with health checks, and Prometheus queries the Consul API to find healthy targets.

A typical setup:

scrape_configs:
  - job_name: 'consul-services'
    consul_sd_configs:
      - server: 'consul.service.consul:8500'
        datacenter: 'dc1'
    relabel_configs:
      - source_labels: [__meta_consul_service]
        action: replace
        target_label: service
      - source_labels: [__meta_consul_node]
        action: replace
        target_label: node
      - source_labels: [__meta_consul_service_port]
        action: replace
        regex: (.+)
        replacement: ${1}
        target_label: __address__

When a service registers with Consul, Prometheus discovers it. When it deregisters or fails its health check, Prometheus stops scraping it. This works well in environments where services manage their own lifecycle and register/deregister on startup and shutdown.

Consul discovery is especially useful for organizations that have invested in Consul as their service mesh or service registry backbone.

File-based discovery for simple setups

Not every infrastructure needs Kubernetes or Consul. For smaller teams, simpler deployments, or development environments, file-based service discovery is practical and transparent.

Create a JSON or YAML file with your targets:

- targets:
    - '192.168.1.10:9100'
    - '192.168.1.11:9100'
  labels:
    job: 'node-exporter'
    environment: 'production'
    region: 'us-east-1'
- targets:
    - 'app-01.internal:9090'
  labels:
    job: 'app-metrics'
    environment: 'staging'

Then configure Prometheus to reload this file periodically:

scrape_configs:
  - job_name: 'file-sd'
    file_sd_configs:
      - files:
          - '/etc/prometheus/targets/*.yml'
        refresh_interval: 30s

Now, when your deployment pipeline updates the targets file, Prometheus picks up the changes within 30 seconds. No reload needed. This approach works well in CI/CD pipelines: your deployment step generates target files based on current infrastructure state, and Prometheus discovers them automatically.

File-based discovery scales reasonably well up to hundreds of targets. Beyond that, consider moving to a more sophisticated discovery mechanism.

Using labels to set job and environment

Labels are how you attach context to your metrics. Rather than relying solely on the job name in your scrape config, use relabel rules to build a rich set of labels that reflect your infrastructure organization.

Here's a pattern that works across different discovery methods:

scrape_configs:
  - job_name: 'services'
    kubernetes_sd_configs:
      - role: pod
    relabel_configs:
      # Keep only pods we want to scrape
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: 'true'
      # Set the job from a pod label
      - source_labels: [__meta_kubernetes_pod_label_app]
        action: replace
        target_label: job
      # Set environment from namespace
      - source_labels: [__meta_kubernetes_namespace]
        action: replace
        target_label: environment
      # Keep the pod name for debugging
      - source_labels: [__meta_kubernetes_pod_name]
        action: replace
        target_label: pod
      # Add cluster name (useful for multi-cluster setups)
      - action: replace
        replacement: 'cluster-1'
        target_label: cluster

With these labels in place, you can write clean PromQL queries: up{environment="production", job="api"} tells you instantly about the state of your production API service. When you aggregate metrics across multiple services, the environment and cluster labels keep things organized and queryable.

Avoiding hard-coded targets

Hard-coded targets in your Prometheus config feel safe initially but create maintenance burden. As your infrastructure grows, keeping the config in sync with reality becomes a full-time job.

The anti-pattern looks like this:

scrape_configs:
  - job_name: 'production'
    static_configs:
      - targets:
          - '10.0.1.5:9090'
          - '10.0.1.6:9090'
          - '10.0.1.7:9090'  # Someone will forget to update this when the IP changes

Instead, let your infrastructure be the source of truth. Whether that's Kubernetes, EC2 tags, Consul registrations, or a deployment-managed file, the targets should derive from your actual running systems, not from manual configuration entries.

Benefits of this approach:

• New instances appear automatically—no deployment required
• Removed or failed instances disappear automatically—no stale metrics
• Configuration becomes a template, not a static list
• Your monitoring code mirrors your infrastructure code

When you need to exclude a specific instance from scraping, use relabel rules to filter it out based on metadata (tags, annotations, labels) rather than removing it from a static list.

Common mistakes

Forgetting to set scrape intervals appropriately

Service discovery can discover targets faster than Prometheus can scrape them. If you're discovering hundreds of new targets and your scrape interval is 10 seconds, you might overwhelm your scrape targets or Prometheus itself. Start with a reasonable global scrape interval (30-60 seconds for most workloads) and adjust based on your infrastructure size and resource constraints.

Ignoring label cardinality

When relabeling, be careful not to create unbounded label values. For example, if you label with the full pod IP address as a label value, and you're creating dozens of pods per minute, you're creating high cardinality metrics that tank Prometheus performance. Use labels like pod name, namespace, and service—not ephemeral identifiers.

Misconfiguring relabel regex

Relabel regex syntax trips up many engineers. The regex must match the entire source value, not just part of it. If you want to extract a port from an address, you need to account for the whole string. Test your regex expressions thoroughly, and use the Prometheus UI's service discovery page to verify what's being discovered.

Missing authentication for discovery APIs

EC2 and Consul discovery endpoints often live in restricted networks or require credentials. Forgetting to configure access keys, service accounts, or RBAC rules means Prometheus can't discover targets at all, and you'll see cryptic errors in the logs. Use your infrastructure's native authentication mechanism: IAM for EC2, service accounts for Kubernetes, tokens for Consul.

Key takeaways

• Service discovery automates target management across dynamic infrastructure and eliminates configuration drift
• Kubernetes SD, EC2 SD, Consul SD, and file-based SD each solve different infrastructure patterns—pick the one that matches your architecture
• Relabel rules transform discovery metadata into queryable labels that reflect your operational structure
• Avoid hard-coded targets; let your infrastructure's source of truth drive what Prometheus monitors
• Pay attention to refresh intervals, authentication, and regex complexity to keep discovery reliable and performant
• Use the Prometheus UI's service discovery page to debug discovery issues and verify what's being found