Monitoring consumer lag: real-world lessons and examples

Monitoring consumer lag: real-world lessons and examples. The single most important Kafka metric.

Why consumer lag is the metric that matters most

You wake up to find your real-time dashboard frozen at yesterday's data. Messages are flowing into Kafka, but your consumer group isn't processing them. You check CPU, memory, disk — all green. What went wrong? The answer usually hides in a single number: consumer lag. Monitoring consumer lag is how we catch problems before they compound into data loss or SLA breaches. In this guide, we'll walk through what lag actually means, why it blinds you when hidden in aggregates, and how to instrument it so you catch problems fast.

What lag means — The gap between the latest offset and the consumer's committed offset

Consumer lag is the distance between where your consumer group should be reading and where it actually is. More formally: it's the difference between the latest offset in a partition and the last offset your consumer has committed.

Think of Kafka partitions as a log. Messages arrive with sequential offsets: 0, 1, 2, 3, and so on. Your consumer reads message 0, processes it, then commits offset 0 to mark "I've handled everything up to here." If the latest offset in the partition is 1000 but your consumer has only committed offset 850, your lag is 150.

Why does this matter? Lag tells you how far behind real-time your application is. A lag of 0 means you're caught up. A lag of 100,000 means you're chasing a backlog that could take minutes or hours to clear—or may never clear at all.

// Simplified mental model
const consumerLag = latestOffset - committedOffset;
// If latestOffset = 1000 and committedOffset = 850
// consumerLag = 150

In production systems we've built, we've seen lag creep silently from 0 to 10,000 in under an hour—usually because a single deployment starved a consumer thread of resources or introduced a bug that made processing 10x slower.

Growing lag — Consumers can't keep up — or they're stuck

Lag grows in two scenarios: either your consumer is slow, or it's stopped entirely.

Scenario 1: Slow processing

Your consumer successfully commits offsets, but new messages arrive faster than you can process them. This happens when:

• You deployed slower code (a new database query that wasn't indexed, for example).
• Downstream services are experiencing latency (a REST API call that used to take 50ms now takes 500ms).
• You didn't scale horizontally to match message volume growth.

Lag will grow steadily. You'll see it climb from 0 to 1,000 to 10,000 over minutes or hours. The good news: if processing speed recovers, lag will eventually stabilize as your consumer catches up.

Scenario 2: Stuck consumer

Your consumer stops committing offsets entirely. The partition keeps accumulating messages, but your consumer hasn't moved in an hour. This happens when:

• The consumer process crashed silently.
• It's deadlocked waiting for a lock or database connection.
• It's stuck in an infinite retry loop.
• The partition leader elected a new replica, and your consumer lost its connection.

In this case, lag will climb without bound. We've seen it reach millions in customer deployments. Unlike slow processing, it won't self-heal.

One real example: we deployed a consumer that called an external API without a timeout. That API went down. The consumer hung, waiting forever, and lag climbed to 500,000+ messages in 20 minutes. We only caught it because we had per-partition lag monitoring wired to an alert.

Measure per partition — One stuck partition hides inside healthy totals

Here's the mistake we see most often: engineers aggregate lag across all partitions of a consumer group and alert on the total. This works until it doesn't.

Imagine a consumer group consuming from a topic with 10 partitions. Nine are healthy, committed offset 900, latest offset 1000—lag of 100 each. But partition 3 is stuck at offset 500, and the latest offset is 2500. That partition alone has lag of 2,000. When you average across the group, you get: (900 + 900 + 900 + 500 + 900 + 900 + 900 + 900 + 900 + 900) / 10 = 890. A healthy average. The alert doesn't fire. Data piles up in partition 3 while you sleep.

We learned this the hard way in a ride-hailing system. A consumer processing geolocation updates crashed silently on one partition. Our aggregate lag metric stayed quiet because nine other partitions kept working fine. Orders started failing because location data was stale. The root cause: we weren't tracking per-partition lag.

Always measure lag per partition, not just per consumer group.

Tools — Burrow, Kafka UI, Prometheus exporters or your cloud provider's metrics

You need visibility into lag. Several tools exist; your choice depends on scale and infrastructure.

Burrow (LinkedIn's open-source lag monitor)

Burrow is purpose-built for lag monitoring. It fetches committed offsets and latest offsets independently, computes lag per partition, and surfaces status. It exposes metrics via HTTP and integrates with Prometheus.

Strengths: Decoupled from your consumer code. Works across consumer groups and clusters. Gives you lag history and trends.

Weakness: Another service to deploy and operate.

Kafka UI (community-maintained UI)

If you run Kafka UI for cluster exploration, it already shows lag per partition and per consumer group. It's visual, so good for dashboards and debugging. But it's not ideal for automated alerting—you'd still need to export metrics to Prometheus or similar.

Prometheus exporters

Tools like kafka-exporter or strimzi-kafka-exporter expose Kafka metrics (including consumer lag) in Prometheus format. You scrape them into Prometheus, then build alerts in Alertmanager.

// Example: Prometheus scrape configuration
// global section might include:
scrape_configs:
  - job_name: 'kafka-exporter'
    static_configs:
      - targets: ['kafka-exporter:9308']
    scrape_interval: 30s

Cloud provider metrics

If you use AWS MSK, Confluent Cloud, or Azure Event Hubs, they expose lag metrics natively. AWS MSK shows lag in CloudWatch. Confluent Cloud has lag in its control plane API. This is often the simplest path if you're already cloud-native—no extra tooling to deploy.

We typically recommend this approach in production: export lag to your existing observability stack (Prometheus, Datadog, New Relic, etc.) and create per-partition alerts. Alert on absolute lag (e.g., "lag > 50,000 for partition X") and rate of lag growth (e.g., "lag increased by 10,000 in 5 minutes").

Common mistakes

• Aggregating lag across partitions. A stuck partition disappears into the average. Always track per-partition lag and alert on max lag per partition.
• Ignoring lag that's growing predictably. If lag grows by 1,000 messages per day because you're ingesting data faster than you process it, that's a capacity problem, not an outage. Set thresholds relative to your SLA, not just "any growth."
• Confusing lag with latency. Lag is a volume metric. Latency is how long one message takes to process. You can have zero lag and high latency (processing is fast, but each message takes 5 seconds). Monitor both.
• Assuming zero lag is always good. If lag stays at zero and your consumer is thrashing the CPU trying to keep up, that's a scaling problem. Lag of 100–1,000 might be healthier if it means you're batching efficiently.
• Not accounting for rebalancing. When a consumer joins or leaves the group, Kafka rebalances. Lag may spike temporarily as partitions reassign. Don't alert on temporary spikes; use a time window of at least 2–3 minutes.

Key takeaways

• Consumer lag is the gap between the latest offset in a partition and the last offset your consumer committed. It's the single most important metric for real-time health.
• Growing lag signals either slow processing or a stuck consumer. Both are urgent, but only the latter will never self-heal.
• Always measure lag per partition. Aggregate metrics hide individual problems.
• Use Burrow, Kafka UI, Prometheus exporters, or cloud-native metrics to surface lag. Export to your alerting system and set thresholds based on your SLA.
• Lag spikes during rebalancing are normal. Give your consumer 2–3 minutes to settle before alerting on lag.