Offsets and commits: a practical Kafka & WebSockets guide

Introduction

You've built a Kafka consumer that processes real-time events from your WebSocket clients—orders, notifications, user interactions. Everything works fine until 2 AM when your consumer crashes mid-flight. When it restarts, which messages does it re-read? The answer lives in offsets and commits: the mechanism that tracks your consumer's position in a partition and determines its resilience to failure. In this guide, we'll walk through offsets and commits: a practical Kafka & WebSockets guide so you can confidently build systems that don't lose or duplicate critical events.

The offset — A position in a partition

An offset is simply a number: the position of a message within a Kafka partition. Partitions are ordered, and every message gets a unique offset starting from 0. When your consumer reads messages, Kafka needs to know where to start next time. That's where offsets matter.

Think of a partition like a book. If you read pages 1–50, your bookmark is at position 50. If the book falls and you pick it up tomorrow, your bookmark tells you exactly where to resume. Kafka works the same way. Your consumer group maintains an offset per partition—a checkpoint that says "we've processed everything up to message N."

Here's what that looks like under the hood:

// Partition 0 has 1000 messages with offsets 0–999
// Your consumer has processed messages 0–456
// Current committed offset: 456
// Next message to read: 457

const consumer = kafka.consumer({ groupId: 'order-processor' });
await consumer.subscribe({ topic: 'orders', fromBeginning: false });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    console.log(`Partition ${partition}, Offset ${message.offset}: ${message.value}`);
    // Each message object includes its offset
  },
});

When you create a consumer group, Kafka automatically assigns partitions to consumers in your group. All consumers in the same group coordinate through offsets stored in a special topic called __consumer_offsets. If you have multiple instances of your service consuming the same topic, each one tracks offsets independently per partition. If one crashes, another instance can pick up from where it left off—but only if the offset was committed.

Commit after processing — Commit only after the work is done to avoid losing events

Committing an offset tells Kafka: "I've finished processing messages up to this point. Remember it." Without a commit, Kafka doesn't know you've done the work. If your consumer crashes, restarted instances don't know where you were and will re-read from the last committed offset.

The rule is simple: only commit after your processing completes successfully. If you commit before processing and the processing fails, you've lost the event. If you process successfully but crash before committing, you'll see the event again—but that's the lesser evil.

Most KafkaJS consumers default to auto-commit, which periodically saves your progress. That's convenient for low-risk scenarios but dangerous for production systems handling payments, inventory, or critical state. Here's a real scenario:

Your service reads an order event, charges the customer's card via an external payment API, and updates your database. If you auto-commit after reading but before the charge completes, and the charge fails due to a network timeout, your consumer will skip that order when it restarts. The customer never gets charged, but your system forgot the order existed.

The safe pattern: disable auto-commit and commit manually after the full pipeline succeeds.

const consumer = kafka.consumer({ groupId: 'payment-processor' });
await consumer.subscribe({ topic: 'orders' });

await consumer.run({
  autoCommit: false, // Critical: don't auto-commit
  eachMessage: async ({ topic, partition, message }) => {
    try {
      const order = JSON.parse(message.value.toString());
      
      // Step 1: Process the message
      const chargeResult = await chargeCard(order.customerId, order.amount);
      
      // Step 2: Update local state
      await db.orders.update(order.id, { status: 'charged', transactionId: chargeResult.id });
      
      // Step 3: Only then commit—confirm Kafka that we're done
      await consumer.commitOffsets([
        {
          topic,
          partition,
          offset: (BigInt(message.offset) + 1n).toString(),
        },
      ]);
      
      console.log(`Order ${order.id} processed and committed`);
    } catch (error) {
      console.error(`Failed to process order: ${error.message}`);
      // Don't commit; next restart will retry this message
    }
  },
});

Notice the offset passed to commitOffsets: it's message.offset + 1. You're telling Kafka to commit up to and including the current message, so next time you start, you'll begin at the next unprocessed message.

Expect reprocessing — Crash after processing but before commit? You'll see the event again

This is the hard truth of distributed systems: your consumer will sometimes process a message twice. If your code charges a customer, processes an order, updates the database—and then your process crashes before committing—when it restarts, Kafka will deliver that same message again. It doesn't know you already processed it.

This isn't a bug; it's a design choice. Kafka trades one-time delivery guarantees for at-least-once delivery: every message will be delivered at least once, but possibly more than once. The alternative would be to commit before processing, which guarantees one-time delivery but risks losing events if processing fails.

The practical consequence: your message handlers must be idempotent. If you process the same order message twice, the second time should be harmless or result in the same state.

// Non-idempotent handler—dangerous
async function handleOrder(order) {
  // What if this runs twice? The customer gets charged twice.
  await chargeCard(order.customerId, order.amount);
  await db.orders.insert(order);
}

// Idempotent handler—safe
async function handleOrder(order) {
  // Check if we've already processed this order
  const existing = await db.orders.findById(order.id);
  if (existing) {
    console.log(`Order ${order.id} already processed`);
    return; // Silently succeed; the message is now safe to skip
  }
  
  // First time seeing this order
  const charge = await chargeCard(order.customerId, order.amount);
  await db.orders.insert({
    ...order,
    transactionId: charge.id,
    processedAt: new Date(),
  });
}

Idempotency doesn't mean doing nothing the second time; it means the result is the same. A well-designed order handler might check for an existing transaction ID and bail early. A notification handler might check if the message was already delivered. The key is: re-running the handler should never change the outcome.

Manual commits for control — Disable auto commit when correctness matters

KafkaJS gives you two commit strategies:

1. Auto-commit (default): the consumer automatically commits every few seconds or after processing N messages. Fast, simple, and risky.
2. Manual commits: you explicitly call commitOffsets() when ready. Slower but precise.

For non-critical systems (analytics, logging, telemetry), auto-commit is fine. For anything affecting money, state, or user trust—payments, inventory, notifications—use manual commits.

Here's how to set up a production-grade consumer with manual commits and a retry backoff:

const consumer = kafka.consumer({
  groupId: 'critical-orders',
  sessionTimeout: 30000, // 30 seconds; increase for slower processing
});

await consumer.subscribe({ topic: 'orders' });

await consumer.run({
  autoCommit: false,
  autoCommitThreshold: 1, // Commit every 1 message (ensures prompt tracking)
  eachMessage: async ({ topic, partition, message }) => {
    const maxRetries = 3;
    let attempt = 0;
    
    while (attempt < maxRetries) {
      try {
        const order = JSON.parse(message.value.toString());
        await processOrder(order);
        
        // Success: commit and break
        await consumer.commitOffsets([
          {
            topic,
            partition,
            offset: (BigInt(message.offset) + 1n).toString(),
          },
        ]);
        
        console.log(`Order ${order.id} committed at offset ${message.offset}`);
        break;
      } catch (error) {
        attempt++;
        console.warn(`Order processing failed, attempt ${attempt}/${maxRetries}: ${error.message}`);
        
        if (attempt < maxRetries) {
          // Exponential backoff: 1s, 2s, 4s
          const delayMs = 1000 * Math.pow(2, attempt - 1);
          await new Promise(resolve => setTimeout(resolve, delayMs));
        } else {
          // Max retries exceeded; log and skip to avoid blocking
          console.error(`Order ${order.id} failed permanently after ${maxRetries} attempts`);
          await consumer.commitOffsets([
            {
              topic,
              partition,
              offset: (BigInt(message.offset) + 1n).toString(),
            },
          ]);
          // Optionally send to a dead-letter queue for manual review
        }
      }
    }
  },
});

Notice the sessionTimeout setting. If your message processing takes longer than the session timeout, Kafka will consider your consumer dead and rebalance the partitions to another instance. That's bad—you'll duplicate work. Increase it if your processing is slow.

Common mistakes

• Committing too early: If you commit before calling your external APIs or database writes, a failure means you've lost the message. Commit only after the entire handler succeeds.
• Forgetting idempotency: Reprocessing is guaranteed. If your handler isn't idempotent, you'll duplicate charges, send duplicate emails, or create duplicate records. Always design for replay.
• Setting session timeout too low: Auto-commit happens during the session. If your processor is slow and the timeout expires, Kafka will kick you out of the consumer group mid-processing, causing a rebalance and duplicates.
• Not handling commit failures: commitOffsets() can fail if the broker is unavailable. Wrap it in try-catch and decide your strategy: retry, log, or skip the commit and let reprocessing handle it on restart.
• Mixing auto-commit and manual commits: Choose one per consumer. Mixing them leads to unpredictable behavior and race conditions.

Key takeaways

• An offset is your consumer's position in a partition; committing saves that position so you know where to resume after a crash.
• Always commit after processing succeeds, never before. This trades occasional reprocessing for guaranteed no data loss.
• Expect reprocessing: every message will be delivered at least once. Design your handlers to be idempotent so reprocessing is safe.
• Disable auto-commit for production systems handling critical data. Use manual commits for full control and visibility.
• Idempotency isn't free—it requires database lookups, unique constraints, or state checks. Factor it into your design from the start.
• Test crash scenarios: kill your consumer mid-processing and verify that restarting resumes correctly and doesn't duplicate work.