Monitoring Kafka: a practical guide for developers

Kafka is a beast. Your team deployed it six months ago to handle order events, and now it powers half your microservices. But when a consumer falls behind, you have no idea why—or even that it's happening. Monitoring Kafka: a practical guide for developers walks you through the metrics that matter, the tools that work, and the alerts that keep your message pipelines healthy.

Why Kafka monitoring is different

Kafka sits between your producers and consumers, processing millions of messages per minute. Unlike a traditional database or HTTP service, Kafka's health isn't just about uptime. A broker can be running perfectly while a consumer hangs three days behind. A topic can fill your disk in hours. Network saturation can degrade without a single error log.

The stakes are real: a lagging consumer means delayed order processing, stale cache data, or analytics that never arrive. That's why we need to monitor three layers: brokers, topics, and consumers. Each has its own metrics, tools, and failure modes.

Broker metrics — JMX exporter for under-replicated partitions and request rates

Your Kafka brokers emit metrics via JMX. We use the Prometheus JMX exporter to scrape them. This gives us visibility into replication health, request throughput, and resource contention.

The most critical broker metrics are:

• Under-replicated partitions (URP): If a partition has fewer in-sync replicas (ISRs) than configured, you have durability risk. One broker dies and you lose data.
• Request rate and latency: How fast brokers process produce and fetch requests. Spikes here signal load or garbage collection pauses.
• Offline partitions: A partition with zero replicas online is completely unavailable. This is a page-worthy alert.
• Network bytes in/out: Saturated network means slow replication and consumer fetches.

Set up the JMX exporter as a Java agent on each broker. Here's a minimal broker configuration:

server:
  port: 5556
lowercaseOutputName: true
lowercaseOutputLabelNames: true
whitelistObjectNames:
  - "kafka.server:type=ReplicaManager,name=*"
  - "kafka.server:type=BrokerTopicMetrics,name=*"
  - "kafka.network:type=RequestMetrics,name=*,clientSoftwareName=*"
  - "kafka.server:type=KafkaRequestHandlerPool,name=*"

Once JMX metrics flow into Prometheus, add these alerts to your rule file:

- alert: KafkaUnderReplicatedPartitions
  expr: kafka_server_replicamanager_underreplicatedpartitions > 0
  for: 5m
  annotations:
    summary: "Broker {{ $labels.instance }} has under-replicated partitions"

- alert: KafkaOfflinePartitions
  expr: kafka_controller_kafkacontroller_offlinepartitionscount > 0
  for: 1m
  annotations:
    summary: "{{ $value }} partitions are offline on broker {{ $labels.instance }}"

Request rate metrics help you spot hot partitions and broker load imbalances. If one broker's produce rate is 10x higher than others, partitions aren't balanced across the cluster. In production e-commerce systems, we've caught runaway producer clients sending 100k msgs/sec to a single topic partition this way.

Consumer lag — kafka-exporter or Burrow

Consumer lag is the distance between the latest offset in a topic and the offset a consumer group has committed. A lag of zero means your consumer is caught up. A lag of 1 million means it's way behind.

We typically use two tools. The simpler option is kafka-exporter, a lightweight standalone scraper. The more robust option is Burrow, LinkedIn's consumer lag monitoring system. Burrow evaluates lag over time windows and generates status (OK, warn, stop) rather than raw numbers.

Deploy kafka-exporter as a sidecar. It exposes metrics like:

kafka_consumer_lag{group="order-processor",topic="orders",partition="0"} 1250
kafka_consumer_lag{group="order-processor",topic="orders",partition="1"} 890
kafka_consumer_lag_sum{group="order-processor",topic="orders"} 2140

The metric kafka_consumer_lag_sum shows total lag across all partitions in a topic. If your order-processor consumer group has 2140 messages of lag, that's typically acceptable for batch processors but unacceptable for real-time pipelines.

Burrow goes deeper. It tracks lag trends, detects when a consumer stops committing offsets (stalled), and alerts based on status rather than raw thresholds. A Burrow status endpoint returns:

{
  "status": "ERR",
  "statusCode": 3,
  "partitions": [
    {"topic": "orders", "partition": 0, "current_lag": 4200, "status": "ERR"}
  ],
  "totalLag": 12000,
  "maxLag": 4200
}

If you're running five consumer groups across thirty topics, Burrow scales. kafka-exporter works fine for small clusters or specific consumer groups you need to watch closely.

Alerts — Lag growing for 10 minutes

Raw lag numbers are noise. A consumer that's 50k messages behind at 2am during a batch job is fine. A consumer that's been growing by 1k msgs/min for ten minutes straight is a problem—the consumer is crashing and restarting, or it's processing slower than messages arrive.

Build an alert that fires when lag increases over a time window. This is more useful than a static threshold:

- alert: KafkaConsumerLagGrowing
  expr: |
    rate(kafka_consumer_lag_sum{topic="orders"}[5m]) > 0
    and
    (kafka_consumer_lag_sum{topic="orders"} - kafka_consumer_lag_sum{topic="orders"} offset 10m) > 1000
  for: 2m
  annotations:
    summary: "Consumer group {{ $labels.group }} lag on {{ $labels.topic }} growing for 10 minutes"
    description: "Lag increased by {{ $value }} messages in last 10 minutes"

This alert fires when lag is increasing (positive rate over 5m) AND absolute lag has grown more than 1000 messages in the past 10 minutes. In a ride-hailing system, this catches the case where your location indexer consumer crashes and messages pile up in the queue.

Pair this with a Grafana dashboard that shows lag over time. Plot lag on the Y-axis and time on the X-axis for each consumer group. Add a horizontal reference line at your SLA (e.g., "lag must stay under 5000 for order processing"). When lag breaches the line, you see it immediately.

Capacity — Disk usage and network throughput

Kafka brokers write every message to disk. A three-broker cluster with 1TB retention policy can fill 3TB of storage. If you don't monitor disk usage, your brokers will eventually run out of space and fail.

Network throughput is equally critical. Kafka replicates partitions across brokers. A broker with 500Mbps network saturation can't keep up with replication or consumer fetches.

Scrape these metrics from the node_exporter on each Kafka broker machine:

• node_filesystem_avail_bytes — Available disk space on the data partition
• node_network_transmit_bytes_total — Bytes sent (in and out)
• node_filesystem_size_bytes — Total disk capacity

Create alerts based on thresholds:

- alert: KafkaDiskUsageHigh
  expr: |
    (1 - (node_filesystem_avail_bytes{mountpoint="/var/kafka-logs"}
    / node_filesystem_size_bytes{mountpoint="/var/kafka-logs"})) > 0.85
  for: 5m
  annotations:
    summary: "Kafka broker {{ $labels.instance }} disk {{ $value | humanizePercentage }} full"

- alert: KafkaNetworkSaturation
  expr: |
    rate(node_network_transmit_bytes_total{device="eth0"}[5m]) / 1e9 > 0.7
  for: 10m
  annotations:
    summary: "Broker {{ $labels.instance }} network transmit near saturation"

In a trading system, we once had a broker hitting 95% disk and didn't notice for six hours because no one was looking. It started rejecting produce requests, orders weren't logged, and the reconciliation team caught it when end-of-day numbers didn't match. Now we alert at 80% and run a daily check.

Capacity planning is often ignored. Ask yourself: at current growth, how many days until we fill this disk? If it's less than 30 days, add more storage or increase log compaction aggressiveness. For throughput, measure your peak bytes/sec rate and ensure each broker's network can handle 3x that (to account for replication and consumer fetches).

Common mistakes

• Alerting on lag alone without context: A consumer 100k messages behind at 10pm might be normal (compaction job, batch processing). Alert on lag growth rate or lag relative to SLA instead.
• Ignoring broker JMX metrics: If a broker silently loses replicas (URP > 0) and you don't alert on it, you have a data loss event waiting to happen. JMX metrics are non-negotiable.
• Not monitoring network saturation: Kafka can produce 10Gbps on a broker with only 1Gbps network. Replication gets stuck, consumers time out, and you see cascading failures.
• Setting alert thresholds without understanding your workload: Batch consumers can have lag spikes. Real-time pipelines cannot. Know your SLAs and alert accordingly.
• Forgetting to monitor Burrow or kafka-exporter itself: If your lag exporter crashes, your dashboard goes blank. Add uptime and error rate alerts for the monitoring tools too.

Key takeaways

• Monitor three layers: brokers (JMX), consumers (lag), and capacity (disk and network).
• Use JMX exporter to track under-replicated partitions and request rates; alert on any URP > 0.
• Deploy kafka-exporter or Burrow to measure consumer lag; use lag growth rate (not raw lag) as your primary alert trigger.
• Alert when lag is growing for 10+ minutes: this signals a stuck or crashing consumer.
• Monitor disk availability (alert at 85%) and network transmit rate; capacity issues kill Kafka clusters silently.
• Build a Grafana dashboard showing lag trends, broker health, and disk usage. Review it during incidents.
• Document your Kafka SLAs (acceptable lag, acceptable URP count) and alert based on violations, not raw thresholds.