Alert on lag, not just errors

Your Kafka consumer is running smoothly—no exceptions in the logs, no failed deployments, no circuit breaker trips. Three hours later, your customers report that orders aren't being processed. You check metrics and realize your consumer fell 500,000 messages behind without triggering a single alert. This is the silent killer of event-driven systems: alert on lag, not just errors. A healthy consumer group can mask severe availability problems beneath a veneer of zero error rates.

I learned this the hard way when building a real-time notification platform. Our CRM system was consuming user events to trigger email campaigns, but our alerts only tracked exceptions and success rates. When a slow database query started blocking writes, the consumer kept pulling messages but processed them slower and slower. By the time we noticed, we'd accumulated a 6-hour backlog. Customers missed time-sensitive offers. The system was never failing—it was just starving.

In this article, I'll walk you through a production-grade alerting strategy that catches lag before it becomes an outage.

Alert on lag growth over time

A single lag measurement is almost useless. A consumer 10,000 messages behind might be healthy if it processes 50,000 messages per minute. The same lag might indicate disaster if throughput is 100 messages per minute. What matters is the trend.

I use a sliding window approach: measure lag every 30 seconds, then compare the 5-minute average to the 15-minute average. If the short-term average is significantly higher, the lag is growing—that's your alert trigger.

typescript


interface LagMetric {
  consumerGroup: string;
  topic: string;
  partition: number;
  currentLag: number;
  timestamp: number;
}

interface LagTrend {
  consumerGroup: string;
  topic: string;
  partition: number;
  lagGrowthRate: number; // messages/second
  average5min: number;
  average15min: number;
  alertTriggered: boolean;
}

class LagTrendAnalyzer {
  private metrics: Map = new Map();
  private readonly windowSize = 30 * 1000; // 30 seconds
  private readonly maxMetricsPerWindow = 300; // 150 minutes of data

  recordLag(metric: LagMetric): void {
    const key = `${metric.consumerGroup}:${metric.topic}:${metric.partition}`;
    if (!this.metrics.has(key)) {
      this.metrics.set(key, []);
    }
    const history = this.metrics.get(key)!;
    history.push(metric);
    
    // Keep only recent data
    const cutoff = metric.timestamp - (this.maxMetricsPerWindow * this.windowSize);
    const filtered = history.filter(m => m.timestamp >= cutoff);
    this.metrics.set(key, filtered);
  }

  analyzeTrend(key: string): LagTrend | null {
    const history = this.metrics.get(key);
    if (!history || history.length < 10) return null;

    const now = Date.now();
    const fiveMinAgo = now - 5 * 60 * 1000;
    const fifteenMinAgo = now - 15 * 60 * 1000;

    const recent5min = history.filter(m => m.timestamp >= fiveMinAgo);
    const recent15min = history.filter(m => m.timestamp >= fifteenMinAgo);

    if (recent5min.length < 2 || recent15min.length < 2) return null;

    const avg5min = recent5min.reduce((sum, m) => sum + m.currentLag, 0) / recent5min.length;
    const avg15min = recent15min.reduce((sum, m) => sum + m.currentLag, 0) / recent15min.length;

    const [group, topic, partition] = key.split(':');
    const growthRate = (avg5min - avg15min) / (10 * 60); // per second

    return {
      consumerGroup: group,
      topic,
      partition: parseInt(partition, 10),
      lagGrowthRate: growthRate,
      average5min: Math.round(avg5min),
      average15min: Math.round(avg15min),
      alertTriggered: growthRate > 100, // alert if lag grows >100 msg/sec
    };
  }
}

This pattern caught a real issue in our ride-hailing platform: a surge in ride bookings caused the pricing consumer to lag by 15%. Without trend-based alerting, we would have waited until throughput hit zero. With trend analysis, we caught it at 2% lag growth and scaled up consumers before customers felt impact.

Adjust the threshold based on your use case. For real-time fraud detection, lag growing faster than 50 messages per second is critical. For daily analytics, 1,000 messages per second growth might be acceptable if you have 8 hours to catch up.

Track time lag, not just message count

Message-count lag is misleading because message size and processing time vary wildly. A consumer processing 100 small events might be fresher than one processing 10 large events.

Time lag answers the right question: "How old is the data I'm currently processing?" It's calculated by taking the timestamp of the oldest unprocessed message and subtracting it from now.

Most Kafka producers embed a timestamp in the message (either the broker timestamp or a custom field). Here's how to track time lag alongside message count:

typescript


interface ConsumerLagMetrics {
  messageLag: number;        // message count
  timeLagSeconds: number;    // age of oldest unconsumed message
  consumerGroup: string;
  topic: string;
  partition: number;
}

class KafkaLagMonitor {
  async getTimeLag(
    consumerGroup: string,
    topic: string,
    partition: number,
    admin: Kafka['admin'],
    consumer: Kafka['consumer']
  ): Promise {
    // Get current consumer offset
    const offsets = await admin.fetchOffsets({ groupId: consumerGroup });
    const consumerOffset = parseInt(
      offsets.find(o => o.topic === topic && o.partition === partition)?.offset ?? '0',
      10
    );

    // Get latest offset
    const topicOffsets = await admin.fetchTopicOffsets(topic);
    const latestOffset = parseInt(
      topicOffsets.find(o => o.partition === partition)?.high ?? '0',
      10
    );

    const messageLag = latestOffset - consumerOffset;

    // Fetch the message at consumer offset to get its timestamp
    const messages = await consumer.fetch({
      topics: [{ topic, partitions: [partition] }],
      fromBeginning: false,
      maxBytes: 1024 * 1024,
    });

    let timeLagSeconds = 0;
    const topicMessages = messages.topics.find(t => t.topic === topic);
    if (topicMessages && topicMessages.partitions.length > 0) {
      const partitionData = topicMessages.partitions.find(p => p.partition === partition);
      if (partitionData && partitionData.messages.length > 0) {
        const oldestUnprocessedMsg = partitionData.messages.find(
          m => parseInt(m.offset, 10) === consumerOffset
        );
        if (oldestUnprocessedMsg && oldestUnprocessedMsg.timestamp) {
          const messageTimestamp = parseInt(oldestUnprocessedMsg.timestamp, 10);
          timeLagSeconds = Math.round((Date.now() - messageTimestamp) / 1000);
        }
      }
    }

    return {
      messageLag,
      timeLagSeconds,
      consumerGroup,
      topic,
      partition,
    };
  }
}

In practice, I've found time lag catches issues that message count misses. Our trading platform's message lag looked fine (only 5,000 behind), but time lag revealed we were 45 seconds stale—old enough that stock prices had shifted and we were processing arbitrage incorrectly. Time lag is your friend.

Alert on consumers with zero throughput

A consumer that's not consuming is the worst-case scenario. It might be crashing, stuck on a poison pill message, or deadlocked waiting for an external dependency. Yet it generates no errors—just silence.

Track throughput per consumer per partition. If throughput is zero for more than 2-3 minutes, alert immediately.

typescript


interface ThroughputMetric {
  consumerGroup: string;
  topic: string;
  partition: number;
  messagesProcessedLast1min: number;
  messagesProcessedLast5min: number;
  lastOffsetUpdate: number;
}

class ThroughputMonitor {
  private lastOffset: Map = new Map();
  private throughputHistory: Map = new Map();

  async checkThroughput(
    consumerGroup: string,
    topic: string,
    partition: number,
    currentOffset: number
  ): Promise<{ healthy: boolean; reason?: string }> {
    const key = `${consumerGroup}:${topic}:${partition}`;
    const previousOffset = this.lastOffset.get(key) ?? currentOffset;
    this.lastOffset.set(key, currentOffset);

    const messagesThisInterval = currentOffset - previousOffset;

    if (!this.throughputHistory.has(key)) {
      this.throughputHistory.set(key, []);
    }

    const history = this.throughputHistory.get(key)!;
    history.push({
      consumerGroup,
      topic,
      partition,
      messagesProcessedLast1min: messagesThisInterval,
      messagesProcessedLast5min: 0,
      lastOffsetUpdate: Date.now(),
    });

    // Keep 5 minutes of data (30-second intervals = 10 data points)
    if (history.length > 10) {
      history.shift();
    }

    const totalLast5min = history.reduce((sum, m) => sum + m.messagesProcessedLast1min, 0);

    // Alert if zero throughput for 2+ minutes
    const recentWithoutMovement = history.filter(
      m => m.messagesProcessedLast1min === 0 && m.lastOffsetUpdate > Date.now() - 2 * 60 * 1000
    );

    if (recentWithoutMovement.length >= 4) {
      return {
        healthy: false,
        reason: `No throughput for 2+ minutes. Total messages last 5min: ${totalLast5min}`,
      };
    }

    return { healthy: true };
  }
}

I've deployed this in our SaaS platform where a single stuck consumer blocks invoice generation. With zero-throughput alerts, we now catch stalls within 2 minutes instead of waiting for customer complaints.

Include partition details in alerts

Generic alerts like "Consumer group lagging" are useless. You need to know which consumer, which topic, and which partition—and you need that info in the alert payload so on-call engineers can act without context-switching.

Here's how I structure alert payload:

typescript


interface AlertPayload {
  severity: 'warning' | 'critical';
  title: string;
  consumerGroup: string;
  topic: string;
  partitions: Array<{
    partition: number;
    currentLag: number;
    timeLagSeconds: number;
    throughputMessagesPerMin: number;
    estimatedCatchupTimeMinutes: number;
  }>;
  affectedEventTypes?: string[];
  recommendedAction: string;
  timestamp: number;
}

function createLagAlert(
  metrics: ConsumerLagMetrics[],
  throughput: number
): AlertPayload {
  const grouped = metrics.reduce((acc, m) => {
    if (!acc[m.topic]) acc[m.topic] = [];
    acc[m.topic].push(m);
    return acc;
  }, {} as Record);

  const topicsAffected = Object.keys(grouped);
  const partitionDetails = metrics.map(m => ({
    partition: m.partition,
    currentLag: m.messageLag,
    timeLagSeconds: m.timeLagSeconds,
    throughputMessagesPerMin: throughput,
    estimatedCatchupTimeMinutes: throughput > 0 
      ? Math.ceil(m.messageLag / throughput / 60)
      : -1,
  }));

  const isCritical = metrics.some(m => m.timeLagSeconds > 300);

  return {
    severity: isCritical ? 'critical' : 'warning',
    title: `${grouped[topicsAffected[0]][0].consumerGroup} lagging on ${topicsAffected.join(', ')}`,
    consumerGroup: metrics[0].consumerGroup,
    topic: topicsAffected[0],
    partitions: partitionDetails,
    affectedEventTypes: topicsAffected,
    recommendedAction: isCritical
      ? 'Scale up consumer instances immediately'
      : 'Monitor closely; consider scaling in next 5 minutes',
    timestamp: Date.now(),
  };
}

This level of detail means an on-call engineer can immediately SSH into a specific consumer instance, check logs for partition 3, and see that it's stuck waiting for Redis. Without partition detail, they're flying blind.

Common mistakes

Alerting on absolute lag, not growth: A consumer 1 million messages behind but processing 100,000 per minute is fine. Focus on whether lag is growing or shrinking.
Ignoring message timestamps: Message count lag is noisy if message sizes vary. Always calculate time lag alongside it.
Setting thresholds without context: "Alert if lag exceeds 50,000" fails because your expected lag depends on traffic patterns. Use percentiles or growth rates instead.
Forgetting to alert on zero throughput: A consumer can be healthy on all metrics except one: actually moving. Monitor it explicitly.
Generic alert titles: "Consumer group lag detected" leaves on-call engineers guessing which consumer, which topic, which partition. Burn the details into the alert message.

Key takeaways

Monitor lag growth over time using sliding windows (5-min vs 15-min averages), not absolute lag values.
Track time lag (how old the oldest unconsumed message is) alongside message count lag for better accuracy.
Alert on zero throughput explicitly—a silent, non-processing consumer is worse than a crashing one because you'll notice the crash.
Include partition-level details in every alert: which partition, current lag, time lag, estimated catchup time, and recommended action.
Set thresholds based on your system's normal behavior and your SLA, not arbitrary numbers.

Alert on lag, not just errors: best practices explained