Auto-commit can lose events: what developers should know

Auto-commit can lose events—and you'll never see it coming

Kafka's default consumer configuration commits offsets automatically on a timer, not after your handler successfully processes each message. This means a crash between the commit and your business logic can skip events entirely, leaving your system inconsistent. I've seen this cost companies real money: duplicate orders that somehow never made it to fulfillment, balance transfers that vanished, ride assignments lost mid-processing.

In this article, I'll walk you through why auto-commit is dangerous for production systems, how crashes exploit this gap, and the practical patterns I use to keep event loss off our incident post-mortems.

Offsets commit on a timer, not after success

By default, Kafka consumers are configured with enable.auto.commit=true and auto.commit.interval.ms=5000. This means every 5 seconds, the consumer automatically commits the offset of the last fetched message—regardless of whether your application has actually processed it.

Think of the offset as a bookmark. When you call consumer.poll(), you fetch messages from the broker. Kafka tracks which messages you've seen. After 5 seconds pass, the broker updates its internal state to say "this consumer is now at offset 1050"—even if you're still processing offset 1045.

Here's a simplified timeline:

1. 13:45:00 - Poll fetches messages 1040-1050
2. 13:45:03 - Application starts processing message 1047
3. 13:45:05 - Auto-commit fires, commits offset 1050 (the highest fetched)
4. 13:45:06 - Application crashes mid-processing message 1047
5. 13:45:08 - Consumer restarts, resumes from offset 1051 (after the last committed)
6. Messages 1047, 1048, 1049, 1050 are lost forever

The interval timer doesn't care about your business logic. It runs independently. Your handler could take 2 seconds or 20 seconds—the timer doesn't know and doesn't wait.

In NestJS, I've seen this pattern:

@Controller('kafka')
export class OrderConsumer {
  constructor(private kafka: KafkaService) {
    this.kafka.subscribe('orders', {
      // Default auto-commit: enabled
      // Default interval: 5000ms
    });
  }

  onMessage(message) {
    // Message 1050 arrives here
    // At 13:45:05, offset 1050 is committed
    // But processing doesn't complete until 13:45:07
    // If we crash at 13:45:06, message is lost
    this.processOrderExpensively(message.value);
  }
}

The handler and the commit timeline are decoupled. That's the core problem.

A crash can skip unprocessed events

Auto-commit doesn't guarantee that the message you just fetched has been handled. It only guarantees that the offset has been recorded. When your application crashes, any unprocessed messages between the last commit and the crash are orphaned.

Let me trace through a real scenario from a trading platform I worked on:

A consumer processes market fill events. Each fill must be persisted to the database and trigger a position update. The Kafka topic has auto-commit enabled with a 5-second interval. At 14:22:10, the consumer polls and fetches fills 5000-5010. By 14:22:15, the offset 5010 is automatically committed. But at 14:22:17, the application crashes while processing fill 5007. The database shows fills 5000-5006 persisted, but 5007-5010 are missing. When the consumer restarts at 14:22:25, it starts at offset 5011. Fills 5007-5010 are gone. Positions are now out of sync with the actual orders filled.

Why doesn't Kafka retain unprocessed messages? Because the offset is the contract between consumer and broker. Once you tell Kafka "I've seen up to offset 5010," Kafka trusts you. It won't replay those messages unless you explicitly request a reset.

In a ride-hailing system, this manifests differently. The consumer processes ride assignments:

async onMessage(kafkaMessage: KafkaMessage) {
  const assignment = JSON.parse(kafkaMessage.value.toString());
  
  // Offset is committed at some point in the next 5 seconds
  // But this operation takes 3 seconds
  await this.rideService.assignDriver(
    assignment.rideId,
    assignment.driverId
  );
  
  // If we crash here (3.2 seconds into processing),
  // the offset was already committed at 5 seconds,
  // and the assignment is stuck in limbo
}

The customer app sees the ride request accepted, the driver app never receives a push notification, and they're both waiting forever.

Use manual commits for business-critical data

The solution is simple: disable auto-commit and commit offsets only after your handler successfully completes. This way, the offset advances only when the message has been processed.

Set enable.auto.commit=false in your consumer config. Now you're responsible for committing. Commit after your business logic succeeds.

In NestJS with Kafka:

@Controller('kafka')
export class OrderConsumer {
  constructor(private kafka: KafkaService, private db: DatabaseService) {}

  async onMessage(message: KafkaMessage, partition: number, offset: number) {
    try {
      const order = JSON.parse(message.value.toString());
      
      // Process first
      await this.db.orders.insert(order);
      await this.inventory.reserve(order.items);
      await this.payment.charge(order.paymentInfo);
      
      // Commit only after success
      await this.kafka.commitOffset(partition, offset + 1);
    } catch (error) {
      // On error, do NOT commit
      // Consumer will retry from the same offset on restart
      console.error('Order processing failed:', error);
      throw error; // Let Kafka client handle retry logic
    }
  }
}

Now the timeline is atomic from the consumer's perspective:

1. 13:45:00 - Poll fetches orders 1040-1050
2. 13:45:03 - Application starts processing order 1047
3. 13:45:06 - Order 1047 is successfully inserted, committed
4. 13:45:07 - Application starts processing order 1048
5. 13:45:09 - Application crashes
6. 13:45:11 - Consumer restarts, resumes from offset 1048 (not 1050)
7. Order 1048 is reprocessed safely

The trade-off is that manual commits are slower. Your throughput might drop slightly because you're waiting for the commit to complete before moving to the next message. But you gain certainty: no message is marked complete until it's actually done.

For highest throughput with safety, batch commits after processing N messages:

async onMessage(message: KafkaMessage, partition: number, offset: number) {
  const order = JSON.parse(message.value.toString());
  
  try {
    await this.processOrder(order);
    this.pendingOffsets.push({ partition, offset: offset + 1 });
    
    // Commit every 50 messages
    if (this.pendingOffsets.length >= 50) {
      for (const pending of this.pendingOffsets) {
        await this.kafka.commitOffset(pending.partition, pending.offset);
      }
      this.pendingOffsets = [];
    }
  } catch (error) {
    // Clear pending on error; don't commit
    this.pendingOffsets = [];
    throw error;
  }
}

This batches commits to reduce latency while still ensuring no message is lost.

Make handlers idempotent either way

Even with manual commits, you should assume your handler might run twice. A network glitch could prevent the commit acknowledgment from reaching your consumer, so you commit again. Or the application crashes after the database write but before the Kafka commit. When the consumer restarts, the same message is processed again.

This is called at-least-once delivery semantics. Kafka guarantees you won't lose a message, but it doesn't guarantee you'll see it only once.

The fix: make your handlers idempotent. Reprocessing the same message should produce the same result.

Bad (not idempotent):

async processTransfer(message) {
  const { fromAccount, toAccount, amount } = JSON.parse(message.value.toString());
  
  // If this runs twice, money transfers twice
  await this.db.accounts.update(fromAccount, { balance: db.raw('balance - ?', amount) });
  await this.db.accounts.update(toAccount, { balance: db.raw('balance + ?', amount) });
}

Good (idempotent):

async processTransfer(message) {
  const { transactionId, fromAccount, toAccount, amount } = JSON.parse(message.value.toString());
  
  // Check if already processed
  const existing = await this.db.transactions.findOne({ id: transactionId });
  if (existing) {
    return; // Already done, skip
  }
  
  // Process and record atomically
  await this.db.transaction(async (trx) => {
    await trx.accounts.update(fromAccount, { balance: trx.raw('balance - ?', amount) });
    await trx.accounts.update(toAccount, { balance: trx.raw('balance + ?', amount) });
    await trx.transactions.insert({ id: transactionId, status: 'completed' });
  });
}

The key is a unique identifier (transactionId) and idempotency tracking. The first time the message is processed, the transaction record is created. The second time, the check finds it and returns early.

Alternatively, use database constraints:

async processTransfer(message) {
  const { transactionId, fromAccount, toAccount, amount } = JSON.parse(message.value.toString());
  
  try {
    await this.db.transaction(async (trx) => {
      // Upsert: insert if new, do nothing if already exists
      await trx.transactions.insert({ id: transactionId, status: 'pending' });
      
      // Now process
      await trx.accounts.update(fromAccount, { balance: trx.raw('balance - ?', amount) });
      await trx.accounts.update(toAccount, { balance: trx.raw('balance + ?', amount) });
      
      await trx.transactions.where({ id: transactionId }).update({ status: 'completed' });
    });
  } catch (error) {
    if (error.code === 'UNIQUE_VIOLATION') {
      // Already processed, this is a retry
      return;
    }
    throw error;
  }
}

The database constraint ensures that even if your application is buggy, you can't process the same transaction twice.

Common mistakes

• Committing in a finally block after consuming the message. If processing throws an error, you still commit, and the message is lost. Always commit only on success.
• Setting auto-commit interval to 0. This doesn't disable auto-commit; it just commits every poll. You still lose the same messages on crash.
• Assuming idempotency is automatic. Kafka doesn't enforce it. Your code must be idempotent. A duplicate message is just a duplicate message unless you detect and skip it.
• Mixing manual and auto-commit. Pick one. Don't flip enable.auto.commit mid-application. Consistency matters.
• Not tracking offset state. If you batch commits, track which offsets are pending. Losing that state means you don't know what to commit next.

Key takeaways

• Auto-commit fires on a timer independent of your handler's success. A crash between commit and processing loses messages.
• Disable auto-commit (enable.auto.commit=false) for any data that matters: orders, payments, positions, assignments.
• Commit offsets only after your business logic completes successfully. On error, don't commit; let the message be retried.
• Expect at-least-once delivery. Build idempotent handlers so reprocessing the same message is safe.
• Use idempotency keys and database constraints to detect and skip duplicate processing.
• Batch commits to reduce latency, but ensure all pending offsets are tracked and committed atomically with your business transaction.