Monitoring Kubernetes workloads: lessons from production

Introduction

Kubernetes gives us powerful abstractions for container orchestration, but it also hides complexity that can bite us in production. Monitoring Kubernetes workloads: lessons from production means moving beyond cluster health metrics to watch what actually matters—the signals that tell us a pod is struggling, a node is saturated, or a deployment is silently failing. We've learned these lessons the hard way, watching production incidents unfold because we weren't instrumenting the right metrics.

After 38 days of Prometheus and Grafana work, we're at the point where generic node metrics feel hollow. A 5% CPU utilization cluster alert means nothing if a single pod is throttled to death. Node memory looks fine until an OOM killer silently terminates a workload. This article focuses on the Kubernetes-specific signals we've found most predictive of real problems.

Pod Restarts and OOM Kills

Pod restarts are your cluster's canary in the coal mine. A pod that restarts occasionally might be handling graceful shutdowns correctly. A pod restarting every few minutes is telling us something is fundamentally wrong, and we're not listening if we don't have that metric visible.

Start by scraping the kube_pod_container_status_restarts_total metric from kube-state-metrics. This gives us the restart count per container.

groups:
  - name: kubernetes_workloads
    interval: 30s
    rules:
      - alert: PodCrashLooping
        expr: rate(kube_pod_container_status_restarts_total[15m]) > 0.1
        for: 5m
        annotations:
          summary: "Pod {{ $labels.pod }} in {{ $labels.namespace }} restarting frequently"
          description: "Pod has restarted {{ $value | humanize }} times per second over 15 minutes"

But restarts alone don't tell us why. OOM kills are the most common culprit in our experience. A pod killed by the kernel's OOM killer shows up differently than a restart triggered by liveness probe failure. We track this using kube_pod_container_last_state_terminated_reason.

      - alert: PodOOMKilled
        expr: kube_pod_container_last_state_terminated_reason{reason="OOMKilled"} == 1
        for: 1m
        annotations:
          summary: "Pod {{ $labels.pod }} OOM killed in {{ $labels.namespace }}"
          description: "Memory limit exceeded; investigate memory leaks or increase resource requests"

In a real scenario we hit: a data processing job was leaking memory gradually. Restarts masked the issue for weeks—they looked like normal pod churn. Only when we started filtering for OOM-specific terminations did we see the pattern. The pod was requested with 512Mi but needed 2Gi. Without that visibility, we kept increasing timeouts and adding retry logic, treating symptoms instead of root causes.

Record the peak memory usage alongside current usage. Use container_memory_working_set_bytes from the kubelet to see how close pods are running to their limits.

      - record: kubernetes:pod_memory_utilization:ratio
        expr: container_memory_working_set_bytes / (kube_pod_container_resource_limits{resource="memory"} > 0) * 100

CPU Throttling

CPU throttling is invisible to most users until latency spikes unexpectedly. Your application is fast locally; in production it's slow. The difference is often CPU throttling on shared nodes, and it shows up nowhere in your error logs.

Kubernetes enforces CPU limits using cgroups. When a container exceeds its limit, it's throttled—not killed, but paused until the next accounting window. This shows up in container_cpu_cfs_throttled_seconds_total.

      - record: kubernetes:pod_cpu_throttled:rate
        expr: rate(container_cpu_cfs_throttled_seconds_total[5m])

      - alert: PodCPUThrottling
        expr: rate(container_cpu_cfs_throttled_seconds_total[5m]) > 0.1
        for: 10m
        annotations:
          summary: "Pod {{ $labels.pod }} is CPU throttled in {{ $labels.namespace }}"
          description: "CPU throttled for {{ $value | humanize }} seconds per second; consider increasing request or limit"

We encountered this with a microservice handling user-facing API traffic. p99 latency was degrading gradually throughout the day as traffic ramped up. The container had a CPU limit of 500m but was consistently requesting only 100m. By noon, it was throttled 40% of the time. The fix was straightforward—increase the request to 300m to reserve more CPU—but we couldn't have diagnosed it without instrumenting throttle metrics.

Pair throttle metrics with container_cpu_usage_seconds_total to understand the gap between requested, used, and limited CPU.

Pending Pods

A pending pod is Kubernetes telling us the scheduler couldn't find a home for the workload. Unlike restarts, pending pods don't restart themselves. They sit in limbo, and the application degrades silently.

Track pods stuck in pending state with kube_pod_status_phase.

      - alert: PendingPods
        expr: kube_pod_status_phase{phase="Pending"} > 0
        for: 5m
        annotations:
          summary: "Pod {{ $labels.pod }} pending for 5+ minutes in {{ $labels.namespace }}"
          description: "Scheduler cannot place pod; check node resources, taints, or node selectors"

The reason a pod stays pending falls into a few categories: not enough resources on any node, taints on nodes that the pod can't tolerate, or a misconfigured node selector. We use a custom recording rule to surface this.

      - record: kubernetes:pending_pods:reason
        expr: |
          group by (namespace, pod, reason) (
            kube_pod_status_phase{phase="Pending"}
            * on(pod, namespace) group_left(reason)
            kube_pod_status_reason{reason!=""}
          )

In production, a deployment scaled automatically but hit the cluster's CPU reserve limit. Pending pods accumulated for 30 minutes before anyone noticed. By then, ingress traffic was timing out. A quick alert threshold of 2-3 pending pods and 5-minute window would have paged us immediately. The fix was reserving more nodes in the cluster, but we couldn't diagnose that without surfacing pending pod metrics prominently in dashboards.

Node Pressure

Nodes report their health status via conditions. When a node runs low on resources, it signals pressure: MemoryPressure, DiskPressure, PIDPressure. These conditions directly affect pod scheduling and eviction decisions.

Use kube_node_status_condition to track node pressure.

      - alert: NodeMemoryPressure
        expr: kube_node_status_condition{condition="MemoryPressure",status="true"} == 1
        for: 5m
        annotations:
          summary: "Node {{ $labels.node }} under memory pressure"
          description: "Kubelet is signaling memory pressure; pods may be evicted"

      - alert: NodeDiskPressure
        expr: kube_node_status_condition{condition="DiskPressure",status="true"} == 1
        for: 5m
        annotations:
          summary: "Node {{ $labels.node }} under disk pressure"
          description: "Kubelet is signaling disk pressure; pods may be evicted or not scheduled"

When a node enters memory pressure, the kubelet begins evicting pods gracefully (if they have lower priority). Lower-priority pods get QoS class Burstable or BestEffort; higher-priority pods get Guaranteed. If you haven't defined pod priority classes and QoS, all pods look equally important to the eviction controller, and evictions become random.

We learned this painfully when a batch processing node ran out of disk. The kubelet started evicting pods indiscriminately. Our database replica pod (which should have been protected) was evicted while the batch job continued running. Node pressure conditions should feed into your runbooks and alert routing, not just be pretty dashboard numbers.

Combine node pressure with kube_node_status_allocatable and actual pod resource requests to predict when pressure will occur.

      - record: kubernetes:node_memory_utilization:ratio
        expr: |
          1 - (
            (kube_node_status_allocatable{resource="memory"} - sum by(node) (
              kube_pod_resource_request{resource="memory"}
            )) / kube_node_status_allocatable{resource="memory"}
          )

Common Mistakes

Only monitoring node-level metrics. A node can report healthy CPU and memory while individual pods are throttled or OOM killed. Shift your monitoring focus down to the workload layer.

Ignoring restart count history. A pod that restarts once every 24 hours might be acceptable; one that restarts every minute is not. Use rate-of-change and time windows in your alert expressions, not raw thresholds.

Not correlating with application metrics. High latency might be CPU throttling, memory pressure, or just slow code. Overlay pod metrics with application performance metrics (request duration, error rates) in the same dashboard to spot the connection.

Setting resource requests too low. The scheduler makes placement decisions based on requests, not actual usage. If all your pods request 10m CPU but use 50m on average, the scheduler will overcommit the cluster and you'll hit throttling and evictions.

Forgetting to enable kube-state-metrics. Many of these metrics come from kube-state-metrics, not the kubelet directly. If you haven't deployed it or haven't set up Prometheus scrape configs for it, these alerts won't fire.

Key Takeaways

• Pod restarts and OOM kills are your first line of defense; filter OOM terminations separately from other crash reasons.
• CPU throttling is invisible to standard monitoring; track container_cpu_cfs_throttled_seconds_total and alert when throttling exceeds 10-20% of time.
• Pending pods indicate scheduler failure; alert on any pod pending for more than 5 minutes and investigate resource requests and node capacity.
• Node pressure conditions (MemoryPressure, DiskPressure) precede pod evictions; surface them prominently and integrate into runbooks.
• Correlate Kubernetes workload metrics with application metrics to distinguish infrastructure issues from code problems.
• Set resource requests to reflect actual usage patterns, not theoretical minimums; the scheduler depends on them for correct placement.