Kafka with Node.js: a practical Kafka & WebSockets guide

Getting started: why the client choice matters

Building a real-time messaging pipeline between Kafka and your Node.js services feels straightforward until you hit production. By Day 26 of the 100-day Kafka & WebSockets series, you've seen how Kafka scales; now we need to wire it into TypeScript services that stay running for months. Kafka with Node.js: a practical Kafka & WebSockets guide means choosing the right client library first—because that decision shapes your entire producer and consumer architecture.

For years, KafkaJS dominated the Node.js ecosystem. It's easy to learn, has solid documentation, and works well in development. However, if you're building for production and relying on active maintenance, vendor support, and long-term stability, KafkaJS has fallen into maintenance mode. The library still works, but critical security patches and protocol updates lag behind. Confluent's official JavaScript client, by contrast, is actively maintained, supports the full Kafka protocol surface, and integrates with Confluent Cloud and on-premises clusters with native authentication and authorization features.

For a billing service consuming orders and a notification service producing alerts, the difference matters. With the official client, you get built-in SASL/SSL support, exactly-once semantics configuration, and Confluent's ecosystem tooling. We recommend it for any service handling payment processing, compliance-sensitive operations, or high-volume transactions.

Understanding the consumer loop: subscribe, process, commit

A Kafka consumer in Node.js follows a predictable pattern: create a consumer instance, subscribe to topics, then enter a loop where you receive messages, decode them, process them, and commit offsets. This loop is where most of your business logic lives, and getting it right means your pipeline won't lose data or duplicate work.

Here's a concrete example: imagine a billing service that listens to an orders topic, calculates invoice amounts, and stores results in a database. The consumer loop handles that workflow:

import { Kafka } from '@confluentinc/confluent-kafka';

const kafka = new Kafka({
  'bootstrap.servers': 'kafka-broker-1:9092,kafka-broker-2:9092',
  'group.id': 'billing',
  'auto.offset.reset': 'earliest',
});

const consumer = kafka.consumer();

async function startConsumer() {
  await consumer.subscribe({ topics: ['orders'] });

  await consumer.run({
    eachMessage: async ({ topic, partition, message }) => {
      try {
        const order = JSON.parse(message.value.toString());
        console.log(`[${partition}] Processing order ${order.id}`);
        
        // Business logic: calculate invoice
        const invoice = await calculateInvoice(order);
        await saveToDatabase(invoice);
        
        // Offset commits automatically with auto.commit=true (default)
      } catch (error) {
        console.error('Failed to process message:', error);
        // Handle poison pill logic here
      }
    },
  });
}

startConsumer().catch(console.error);

Notice how the eachMessage callback receives the entire message object: topic, partition, timestamp, key, value, and headers. Your code decodes the value (usually JSON), processes it, and either succeeds or throws an error. The consumer automatically manages offset commits in the background if auto.commit is enabled, which is the default. This keeps your consumer group tracking where you've read up to, so if the process restarts, it resumes from the last committed offset.

In real systems, your processing might involve multiple steps: enrich the order with customer data, validate against inventory, call a payment gateway, and only then write the invoice. Each step should be idempotent—if a failure occurs and a message is re-processed, the outcome should be identical. Using database transactions and upsert queries instead of plain inserts is the safest pattern.

Graceful shutdown: disconnecting cleanly to avoid data loss

A consumer loop runs indefinitely. When you deploy a new version, scale down, or encounter a hardware failure, the process receives a SIGTERM signal. How you handle that signal determines whether offsets commit cleanly and rebalances complete without leaving your consumer group in a broken state.

Without graceful shutdown, the broker thinks your consumer is dead. It waits for the session timeout (default 45 seconds), then rebalances partitions to other group members. If you were in the middle of processing a message, that message might be processed twice—once by you and once by a failover consumer. Even worse, uncommitted offsets mean you might lose recent messages entirely.

Here's how to handle signals properly:

import { Kafka } from '@confluentinc/confluent-kafka';

const kafka = new Kafka({
  'bootstrap.servers': 'kafka-broker-1:9092',
  'group.id': 'billing',
  'session.timeout.ms': 45000,
});

const consumer = kafka.consumer();
let isShuttingDown = false;

async function startConsumer() {
  await consumer.subscribe({ topics: ['orders'] });

  // Set up signal handlers before starting the loop
  process.on('SIGTERM', async () => {
    console.log('SIGTERM received, starting graceful shutdown...');
    isShuttingDown = true;
    
    try {
      // Commit any pending offsets and leave the group
      await consumer.disconnect();
      console.log('Consumer disconnected cleanly');
      process.exit(0);
    } catch (error) {
      console.error('Error during shutdown:', error);
      process.exit(1);
    }
  });

  await consumer.run({
    eachMessage: async ({ partition, message }) => {
      if (isShuttingDown) {
        return; // Stop processing new messages
      }
      
      try {
        const order = JSON.parse(message.value.toString());
        await processOrder(order);
      } catch (error) {
        console.error('Processing failed:', error);
      }
    },
  });
}

startConsumer().catch(console.error);

The pattern is simple: set a flag when SIGTERM arrives, stop accepting new messages, call consumer.disconnect(), and exit. The disconnect call waits for the current batch of messages to finish processing, commits offsets, and sends a LeaveGroup request to the broker. This triggers an immediate rebalance among remaining consumers, avoiding the session timeout delay.

In Kubernetes, pair this with a preStop hook that gives your service 30 seconds to drain before sending SIGKILL. In systemd, set TimeoutStopSec generously. Testing shutdown behavior in staging is non-negotiable; we've seen production incidents where services exited without disconnecting, causing cascading rebalances.

Concurrency and partition ordering: parallel work with guarantees

Kafka's concurrency model is often misunderstood. A topic can have many partitions, and your consumer group automatically distributes them across instances. Within each partition, messages are ordered and assigned to a single consumer. If you have three consumers and nine partitions, each consumer gets three partitions and processes them serially.

The question is: can we process messages within a single partition in parallel? The answer is nuanced. If your business logic requires strict ordering—for example, crediting a customer's account must happen after debiting it—then no. Process serially within each partition. If ordering doesn't matter, or if you're processing independent items (say, sending emails), then yes, you can parallelize.

Here's an example of ordered processing per partition with optional local concurrency:

async function startConsumer() {
  await consumer.subscribe({ topics: ['orders'] });

  // Process sequentially: one message at a time per partition
  await consumer.run({
    eachMessage: async ({ partition, message }) => {
      const order = JSON.parse(message.value.toString());
      await processOrder(order); // Blocks until complete
    },
  });
}

// Alternative: batch processing with controlled concurrency
async function startConsumerWithBatching() {
  await consumer.subscribe({ topics: ['orders'] });

  const messageQueue = [];
  const batchSize = 10;
  const concurrency = 5;

  await consumer.run({
    eachMessage: async ({ message }) => {
      messageQueue.push(message);

      if (messageQueue.length >= batchSize) {
        const batch = messageQueue.splice(0, batchSize);
        
        // Process batch with controlled concurrency
        await Promise.all(
          batch.map((msg) => processOrder(JSON.parse(msg.value.toString())))
        );
      }
    },
  });
}

The first approach is safest for financial transactions. The second is useful for I/O-bound work like HTTP requests or database queries, where you want to handle multiple messages simultaneously but still commit offsets in order.

A practical real-world scenario: a notification service consuming from a user-events topic. Each event triggers an email or SMS. Events from the same user probably shouldn't be delivered out of order (imagine "confirm your email" arriving after "your email was verified"). You'd process sequentially per partition, but since events from thousands of users are spread across partitions, you achieve high throughput without violating ordering constraints.

Common mistakes

Committing before processing completes: If you manually commit offsets before your database write finishes, a crash between the commit and the write causes data loss. Always commit after successful processing, or use transactions to make the whole flow atomic.

Ignoring rebalancing: When partitions are reassigned, you may receive duplicate messages during the rebalance window. Design your handlers to be idempotent. Using a deduplication table keyed by message ID is a reliable pattern.

Not setting session timeouts appropriately: Default is 45 seconds. If your message processing takes longer, increase session.timeout.ms and heartbeat.interval.ms` proportionally, or split work into smaller tasks.

Blocking the event loop: Long-running synchronous operations (CPU-intensive parsing, heavy calculations) will starve the consumer loop. Offload to worker threads or async tasks.

Missing error handling in eachMessage: If your callback throws an exception, the consumer logs it and continues. You must decide: skip the message, retry with exponential backoff, or send it to a dead-letter topic. There's no one-size-fits-all answer, but silence is dangerous.

Key takeaways

• Use Confluent's official JavaScript client for production Node.js services; KafkaJS is maintained but not actively developed.
• Consumer loops follow the pattern: subscribe, receive messages via eachMessage, decode, process, and commit offsets automatically or manually.
• Graceful shutdown is critical—catch SIGTERM, set a flag to stop accepting messages, call consumer.disconnect(), and exit cleanly to avoid rebalance storms.
• Partition ordering is preserved within a partition but not across partitions; design concurrency around your consistency requirements.
• Always make message processing idempotent and handle errors explicitly; silence breeds production incidents.