Prometheus in Kubernetes: a hands-on guide for engineers

Introduction

Running Prometheus at scale inside Kubernetes is a different beast than managing a standalone instance. Your cluster is dynamic: pods spin up, services appear and disappear, nodes scale in and out. Traditional static scrape configurations fall apart. Prometheus in Kubernetes: a hands-on guide for engineers walks you through the modern approach—using Helm charts, ServiceMonitor resources, and built-in exporters—so your monitoring adapts as fast as your infrastructure does.

Over the last 35 days, we've covered Prometheus fundamentals, PromQL, alerting rules, and dashboard design. Today we lock in the operational picture: how to deploy Prometheus into a Kubernetes cluster, scrape metrics from workloads automatically, collect cluster health data, and size everything so it doesn't become a resource hog that crashes your cluster.

kube-prometheus-stack — Helm chart with Prometheus, Grafana and Alertmanager

The Prometheus community maintains kube-prometheus-stack, a Helm chart that bundles Prometheus, Grafana, Alertmanager, and a collection of preconfigured dashboards and recording rules. It's production-ready and handles most of the wiring you'd otherwise do manually. Rather than deploying these components separately, the stack gives you cohesion: Grafana is pre-configured to talk to Prometheus, Alertmanager is already integrated into your Prometheus config, and you get sensible defaults from day one.

Let's deploy it. First, add the Prometheus community Helm repository:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

Now create a values file to customize the deployment. A typical starting point for a mid-sized cluster looks like this:

prometheus:
  prometheusSpec:
    retention: 15d
    resources:
      requests:
        cpu: 500m
        memory: 2Gi
      limits:
        cpu: 1000m
        memory: 4Gi
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: standard
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 50Gi

grafana:
  adminPassword: "change-me-in-production"
  persistence:
    enabled: true
    size: 10Gi

alertmanager:
  enabled: true
  config:
    global:
      resolve_timeout: 5m
    route:
      receiver: 'null'
    receivers:
      - name: 'null'

Install the chart into a dedicated namespace:

helm install kube-prometheus-stack prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  -f values.yaml

Within a minute or two, you have Prometheus and Grafana running. Port-forward to Grafana to see it:

kubectl port-forward -n monitoring svc/kube-prometheus-stack-grafana 3000:80

Navigate to localhost:3000, log in with admin / your password, and you'll find dashboards already loaded. The stack installs ServiceMonitors for the Kubernetes control plane and node exporters, so metrics are flowing immediately.

ServiceMonitor — Declare scrape targets as Kubernetes resources

In a static Prometheus setup, you edit prometheus.yml and list every target by hostname or IP. In Kubernetes, that's backwards. Pods are ephemeral. You need dynamic discovery, and that's where ServiceMonitor comes in.

A ServiceMonitor is a Kubernetes custom resource that tells Prometheus what to scrape. The Prometheus operator (installed as part of kube-prometheus-stack) watches for ServiceMonitors in the cluster, translates them into scrape configs, and reloads Prometheus without downtime.

Here's a real example: suppose your team runs a payment processing service that exposes metrics on port 8080 at the /metrics endpoint. You've already created a Kubernetes Service for it. Now you write a ServiceMonitor:

apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payment-service-monitor
  namespace: production
spec:
  selector:
    matchLabels:
      app: payment-service
  endpoints:
    - port: metrics
      interval: 30s
      path: /metrics

This tells Prometheus: find all Services in the production namespace with the label app: payment-service, scrape port metrics every 30 seconds at /metrics. The Service definition would look like:

apiVersion: v1
kind: Service
metadata:
  name: payment-service
  namespace: production
  labels:
    app: payment-service
spec:
  selector:
    app: payment-service
  ports:
    - name: metrics
      port: 8080
      targetPort: 8080

When you apply the ServiceMonitor, the Prometheus operator detects it, updates Prometheus's scrape configuration, and your payment service metrics appear in Prometheus—no manual reload, no downtime. If the Service scales up or down, Kubernetes service discovery handles it. This is the automation win that makes Prometheus viable at cluster scale.

Cluster metrics — kube-state-metrics and cAdvisor

Prometheus doesn't know about Kubernetes abstractions out of the box. It can't tell you how many replicas a Deployment should have versus how many are actually running. It can't list all PersistentVolumeClaims or their status. That's where kube-state-metrics comes in.

kube-state-metrics watches the Kubernetes API and exposes metrics for every object: Deployments, StatefulSets, DaemonSets, Pods, Nodes, PVCs, and more. The kube-prometheus-stack Helm chart installs it automatically. Metrics like kube_deployment_status_replicas, kube_pod_status_phase, and kube_persistentvolumeclaim_status_phase let you detect when reality diverges from intent.

Alongside kube-state-metrics, you need cAdvisor (container Advisor), which runs on every node as a kubelet component and exposes container resource usage: CPU, memory, network I/O, disk I/O. You'll query metrics like container_cpu_usage_seconds_total and container_memory_usage_bytes to understand individual pod consumption.

Here's a practical example: you want to alert when a Deployment has not reached its desired replica count for more than 5 minutes. The alert rule uses kube-state-metrics:

apiVersion: monitoring.coreos.com/v1
kind: PrometheusRule
metadata:
  name: deployment-replicas-alert
  namespace: monitoring
spec:
  groups:
    - name: kubernetes
      interval: 30s
      rules:
        - alert: DeploymentReplicasNotReady
          expr: |
            kube_deployment_status_replicas_available{namespace="production"}
            <
            kube_deployment_spec_replicas{namespace="production"}
          for: 5m
          annotations:
            summary: "Deployment {{ $labels.deployment }} has missing replicas"

This rule fires when available replicas fall short of desired replicas for 5 minutes. Combined with other kube-state-metrics, you get full visibility into cluster state.

Resource limits — Prometheus needs memory planning

Prometheus holds all time-series data in memory before flushing to disk. If your cluster has many services, many metrics per service, and a high scrape frequency, Prometheus's memory footprint climbs fast. Underestimate it and your Prometheus pod gets OOMKilled; overestimate and you're paying for capacity you don't use.

A rough mental model: each unique time series (metric name + label combination) consumes about 1-2 KB of memory. If you're scraping 100 services, each exposing 500 metrics, with 10 common labels, you're looking at roughly 100 × 500 × 10 × 1.5 KB ≈ 7.5 GB. Add headroom for query processing, and suddenly you need 10–12 GB allocated.

But you also need to configure retention and WAL (write-ahead log) settings. In the kube-prometheus-stack values file above, we set retention: 15d. Longer retention means more disk space and higher memory pressure. For a busy cluster, 15 days is reasonable; for a larger one, you might drop to 7 days or use separate long-term storage (like Thanos).

The WAL consumes additional disk: by default, roughly 128 MB per WAL segment, and Prometheus keeps a few segments. Set walSegmentSize if you're space-constrained:

prometheus:
  prometheusSpec:
    walSegmentSize: 512MB
    retention: 10d
    resources:
      requests:
        cpu: 500m
        memory: 4Gi
      limits:
        cpu: 2000m
        memory: 8Gi
    storageSpec:
      volumeClaimTemplate:
        spec:
          storageClassName: fast-ssd
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 100Gi

Monitor Prometheus's own memory with process_resident_memory_bytes. Set up an alert if it approaches 80% of its limit. If Prometheus is consistently bumping against limits, you have three levers: reduce scrape frequency, drop unnecessary labels or metrics, or add a second Prometheus instance to shard the workload.

Common mistakes

• Not labeling Services correctly for ServiceMonitor selectors: If your Service doesn't have the label your ServiceMonitor expects, nothing gets scraped. Always verify the label chain: Pod labels → Deployment selector → Service selector → ServiceMonitor selector.
• Scraping too frequently: Setting interval to 5 seconds on 50 targets means 10 scrapes per second. Your infrastructure can handle it, but you're carving away retention life quickly and straining Prometheus. 30 seconds is a sensible default; go lower only if you have a specific reason.
• Ignoring Prometheus's own resource usage: Prometheus is a data-hoarding engine. It's easy to let it consume 90% of a node's memory. Set requests and limits early, then monitor and adjust as data grows.
• Using the wrong metric for alerting on cluster health: Confusing kube-state-metrics (desired vs. actual state) with kubelet metrics (resource consumption) leads to noisy or silent alerts. kube-state-metrics tells you a Deployment is unhealthy; kubelet metrics tell you a pod is CPU-bound.
• Storing Prometheus on ephemeral storage: If your PVC is ephemeral or your node dies, Prometheus loses all historical data. Use a persistent, replicated storage backend. For production, consider Thanos for multi-replica and long-term storage.

Key takeaways

• The kube-prometheus-stack Helm chart bundles Prometheus, Grafana, Alertmanager, and node exporters into one production-ready release.
• ServiceMonitor is a Kubernetes custom resource that dynamically wires services into Prometheus scrape configs, eliminating static configuration.
• kube-state-metrics exposes Kubernetes object state (Deployments, Pods, PVCs) as metrics; cAdvisor exposes container resource usage.
• Plan Prometheus memory usage upfront: roughly 1–2 KB per time series. Set requests and limits, monitor cardinality, and adjust retention as your cluster grows.
• Combine cluster metrics with application metrics for end-to-end observability: know both what's happening in your infrastructure and what your code is doing.