Prometheus Operator: Monitoring best practices explained

Introduction

By day 37 of your monitoring journey, you've likely deployed Prometheus and Grafana to collect and visualize metrics from your Kubernetes cluster. But managing Prometheus configuration files directly—editing scrape configs, reloading, and coordinating changes across teams—doesn't scale. The Prometheus Operator transforms how you manage monitoring by letting you define scrape targets, rules, and Prometheus instances using native Kubernetes resources. In this article, we'll explore Prometheus Operator: Monitoring best practices explained and show you how to leverage ServiceMonitor, PodMonitor, PrometheusRule, and GitOps workflows to build a robust, declarative monitoring stack.

What is the Prometheus Operator?

The Prometheus Operator is a Kubernetes controller that watches custom resource definitions (CRDs) and reconciles them into Prometheus and Alertmanager configurations. Instead of manually editing prometheus.yml and reloading the server, you declare what to monitor using Kubernetes manifests. The operator handles the rest: generating configs, managing ConfigMaps, and orchestrating pod restarts when needed.

This approach brings several advantages:

• Declarative configuration: Monitor resources are defined alongside your applications in Git.
• Automatic discovery: Targets are discovered and registered without manual intervention.
• Team self-service: Application teams can add monitoring for their services without involving platform engineers.
• Version control: All monitoring decisions are auditable and reversible through Git history.

ServiceMonitor: Scraping Kubernetes Services

A ServiceMonitor tells Prometheus which Kubernetes Services to scrape and how to scrape them. It works by selecting Services via label selectors and specifying the port and metrics path.

Consider a Node.js microservice that exposes Prometheus metrics on port 3000 at /metrics. Your deployment looks like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-api
  namespace: production
spec:
  selector:
    matchLabels:
      app: payment-api
  template:
    metadata:
      labels:
        app: payment-api
        version: v1
    spec:
      containers:
      - name: api
        image: payment-api:v1.2.3
        ports:
        - name: metrics
          containerPort: 3000
---
apiVersion: v1
kind: Service
metadata:
  name: payment-api
  namespace: production
  labels:
    app: payment-api
spec:
  selector:
    app: payment-api
  ports:
  - name: metrics
    port: 3000
    targetPort: 3000

Now create a ServiceMonitor to scrape it:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payment-api
  namespace: production
spec:
  selector:
    matchLabels:
      app: payment-api
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics

The Prometheus Operator watches for this ServiceMonitor. It then updates the Prometheus scrape configuration to include the payment-api Service. If the Service has 3 pods, Prometheus automatically discovers and scrapes all three endpoints. When a pod scales down or is replaced, the config updates automatically.

A real-world scenario: your platform team manages a shared Prometheus instance. Application teams create ServiceMonitors in their namespaces; the operator discovers them via namespaceSelector and adds them to the scrape job. Teams deploy monitoring independently, no bottleneck at the platform layer.

PodMonitor: Direct Pod Scraping

Sometimes you need to scrape pods directly rather than through a Service. A PodMonitor selects pods by label and scrapes their metrics endpoints directly.

Imagine a distributed cache cluster where each pod exposes metrics on port 6379. You might not want a Service; you need to scrape each pod independently:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: redis-cluster
  namespace: cache
spec:
  selector:
    matchLabels:
      app: redis
  serviceName: redis
  replicas: 5
  template:
    metadata:
      labels:
        app: redis
        cluster: main
    spec:
      containers:
      - name: redis
        image: redis:7-alpine
        ports:
        - name: metrics
          containerPort: 6379
---
apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: redis-cluster
  namespace: cache
spec:
  selector:
    matchLabels:
      app: redis
      cluster: main
  podMetricsEndpoints:
  - port: metrics
    interval: 15s

The PodMonitor discovers all pods matching the label selector and scrapes them on the specified port. Unlike ServiceMonitor, each pod is a separate scrape target; you get per-pod metrics without aggregation through a Service. This is essential for stateful workloads where pod identity matters.

PrometheusRule: Defining Alerts and Recording Rules

Alerting logic belongs in version control too. PrometheusRule is a CRD for alert rules and recording rules, keeping your alert logic alongside your deployments.

For the payment-api service, you might define rules for error rates and latency:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: payment-api-rules
  namespace: production
spec:
  groups:
  - name: payment-api
    interval: 30s
    rules:
    - alert: PaymentAPIHighErrorRate
      expr: |
        (sum(rate(http_requests_total{job="payment-api", status=~"5.."}[5m]))
         / sum(rate(http_requests_total{job="payment-api"}[5m])))
        > 0.05
      for: 5m
      labels:
        severity: critical
        team: payments
      annotations:
        summary: "Payment API error rate above 5%"
        runbook: "https://wiki.example.com/payment-api-errors"
    - alert: PaymentAPIHighLatency
      expr: |
        histogram_quantile(0.95,
          sum(rate(http_request_duration_seconds_bucket{job="payment-api"}[5m]))
          by (le)
        ) > 1
      for: 10m
      labels:
        severity: warning
        team: payments
      annotations:
        summary: "Payment API p95 latency above 1 second"
    - record: payment:request_rate:5m
      expr: sum(rate(http_requests_total{job="payment-api"}[5m])) by (status)

The Prometheus Operator watches PrometheusRule resources across all watched namespaces, aggregates them, and writes them into the Prometheus configuration. When you update the rule, the operator detects the change and triggers a Prometheus reload without dropping active queries.

Recording rules like payment:request_rate:5m precompute expensive aggregations, improving query performance for dashboards and alerts.

Managing Prometheus Instances and Configuration

The Prometheus CRD lets you define Prometheus instances themselves as Kubernetes resources. You specify the version, resource requests, storage, and which ServiceMonitors and PrometheusRules it should use:

apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: monitoring
  namespace: monitoring
spec:
  image: quay.io/prometheus/prometheus
  version: v2.48.0
  replicas: 2
  resources:
    requests:
      cpu: 500m
      memory: 2Gi
    limits:
      cpu: 2
      memory: 4Gi
  storage:
    volumeClaimTemplate:
      spec:
        accessModes: [ "ReadWriteOnce" ]
        resources:
          requests:
            storage: 100Gi
  serviceMonitorSelector:
    matchLabels:
      prometheus: main
  podMonitorSelector:
    matchLabels:
      prometheus: main
  ruleSelector:
    matchLabels:
      prometheus: main
  retention: 30d
  retentionSize: "90Gi"
  alertingSpec:
    alertmanagers:
    - namespace: monitoring
      name: alertmanager
      port: web

Notice the serviceMonitorSelector, podMonitorSelector, and ruleSelector. They use label selectors to determine which ServiceMonitors, PodMonitors, and PrometheusRules this Prometheus instance should scrape from. This allows multiple Prometheus instances in the same cluster, each owning a subset of monitoring targets.

In a large organization, you might have a Prometheus instance per team or environment, each with its own retention and resource policy, reducing blast radius and enabling independent scaling.

Version Control and GitOps Integration

The true power of the Prometheus Operator emerges when you version-control all monitoring definitions in Git and use GitOps to keep your cluster in sync.

Organize your repository like this:

With ArgoCD or Flux, a pull request that adds a ServiceMonitor to payment-api is automatically synced to the cluster. If the change breaks alerting thresholds, you review the diff, test in staging, and merge with confidence. Rollback is a Git revert away.

Example workflow:

1. Developer in the payments team adds a new PrometheusRule for a new payment flow.
2. They submit a PR to services/payment-api/monitoring/alert-rules.yaml.
3. Team lead reviews the PromQL expressions and annotations, checks that it aligns with on-call runbooks.
4. PR is merged. ArgoCD detects the change and applies it to the cluster within seconds.
5. The Prometheus Operator updates the Prometheus configuration and reloads it.
6. New alerts are live, auditable, and reversible.

Common Mistakes

Forgetting the selectors: If your Prometheus CRD has no serviceMonitorSelector, it won't scrape anything even if you create perfect ServiceMonitors. Always define label selectors that match your monitors.

Label mismatch: ServiceMonitors select Services by labels. If your Service is missing the expected label, it won't be scraped. Use kubectl get svc --show-labels to verify.

Path and port mismatches: Double-check that the port in your ServiceMonitor matches the port name (not number) in the Service definition. The path must match where your application exposes metrics.

Retention policy forgotten: PrometheusRule changes don't trigger storage cleanup. Set retention or retentionSize on the Prometheus CRD to avoid disk exhaustion.

RBAC gaps: The Prometheus Operator needs permissions to read ServiceMonitors, PodMonitors, and PrometheusRules. If you're using a restrictive service account, grant the right ClusterRoles.

Key Takeaways

• Prometheus Operator replaces manual scrape config management with Kubernetes-native CRDs, making monitoring declarative and auditable.
• ServiceMonitor discovers Services by label and scrapes them; PodMonitor scrapes individual pods directly.
• PrometheusRule lets you version-control alert and recording rules alongside application code.
• The Prometheus CRD defines Prometheus instances with version, resources, storage, and selectors to control which monitors and rules it uses.
• GitOps integration ensures monitoring changes are reviewed, tracked, and reversible through Git history.
• Label selectors are critical; mismatches between your monitors and the resources they target are the most common source of silent failures.
• Multiple Prometheus instances with different selectors enable teams to own their monitoring independently without platform bottlenecks.