Topics are not queues for Kafka & WebSockets developers

I spent my first week with Kafka thinking it was just a faster, fancier message queue. I was wrong, and that misunderstanding cost me hours of debugging in production. Topics are not queues—and once you internalize that difference, Kafka's design starts to make real sense.

The core mental shift

Most of us come from traditional message queue backgrounds: RabbitMQ, AWS SQS, or even simple job queues in Redis. In those systems, a message exists for one purpose: to be consumed once, then deleted. You enqueue a task, a worker picks it up, processes it, and the message vanishes. Clean. Simple. But Kafka works in almost the opposite way.

Kafka topics are event logs, not transient buffers. When you produce an event to a Kafka topic, it doesn't exist to be consumed and forgotten—it exists to be read by potentially many consumers, potentially many times, over a defined retention period. This fundamental difference shapes everything about how you architect systems with Kafka.

Queues delete messages once consumed

In a traditional queue, consumption and deletion are tightly coupled. Here's what that flow looks like:

// Traditional queue behavior (e.g., AWS SQS)
const message = await queue.receiveMessage();
console.log("Processing:", message.body);
await processMessage(message.body);
await queue.deleteMessage(message); // Gone forever

Once your worker acknowledges that message, it's removed from the queue. If another consumer needs that data, you have to re-produce it or fetch it from somewhere else. Queues are designed for at-most-once delivery semantics in simple cases, or at-least-once with careful acking—but the core assumption is: one consumer group, one flow, one outcome.

This works great for task distribution. You have 10 orders to process, 5 workers, and each order disappears after being handled. Perfect.

But what if you later want to analyze all orders from the past month? Or replay processing because you deployed a buggy service? Or add a second consumer group to do real-time analytics while the first group does persistence? With a traditional queue, you're stuck. The messages are gone.

Kafka keeps events for a retention period

Kafka flips the deletion model on its head. Events land in a topic and stay there—on disk, in order—for as long as your retention policy allows. The default retention in many deployments is 7 days, but you can set it to hours, days, weeks, or even indefinite.

// Kafka producer: fire and forget (or not)
const { Kafka } = require('kafkajs');

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092'],
});

const producer = kafka.producer();
await producer.connect();

// This event lives on disk for 7 days (default retention)
await producer.send({
  topic: 'user-events',
  messages: [
    {
      key: 'user-123',
      value: JSON.stringify({
        userId: 123,
        action: 'login',
        timestamp: Date.now(),
      }),
    },
  ],
});

await producer.disconnect();

That event is now written to a topic partition, replicated across your brokers, and stored on disk. It doesn't vanish when consumed. It sits there. A consumer can read it, process it, and move on. The event remains. Another consumer can read the same event tomorrow and get the same data.

This is why Kafka is often used as an event store or event log. It becomes a source of truth. Your analytics service can consume events to build dashboards. Your ML pipeline can consume the same events to train models. Your compliance system can consume them for audit trails. All from the same immutable log.

The retention period is configurable per topic. You might keep high-volume clickstream data for 24 hours but keep financial transaction events for a year. Retention can be based on time, storage size, or even segment age in newer versions of Kafka.

Many consumer groups read independently

Here's where the architecture really diverges from queues. In Kafka, multiple consumer groups can independently consume the same topic, each at their own pace, without interfering with each other.

A consumer group is identified by a group.id. All consumers with the same group.id form a group. Kafka coordinates partition assignment so each partition is handled by exactly one consumer in a group—but different groups maintain separate offset tracking. Group A might be at message 50,000 while Group B is at message 10,000, both reading the same topic.

// Consumer Group 1: Real-time dashboard
const consumer1 = kafka.consumer({ groupId: 'analytics-group' });
await consumer1.subscribe({ topic: 'user-events' });
await consumer1.run({
  eachMessage: async ({ topic, partition, message }) => {
    console.log('Analytics:', message.value.toString());
    // Update real-time metrics
  },
});

// Consumer Group 2: Batch processing (separate group, same topic)
const consumer2 = kafka.consumer({ groupId: 'batch-group' });
await consumer2.subscribe({ topic: 'user-events' });
await consumer2.run({
  eachMessage: async ({ topic, partition, message }) => {
    console.log('Batch processor:', message.value.toString());
    // Write to data warehouse
  },
});

Both groups read from user-events. If analytics-group crashes and falls behind, batch-group keeps chugging. If you deploy a new batch processor and want it to start from the beginning, you can reset the offset for that group. The other group is unaffected.

This is impossible with a traditional queue unless you duplicate the messages or build complex routing logic. In Kafka, it's built in.

I used this pattern in a ride-hailing platform. We had one consumer group for real-time driver matching, another for historical analytics, another for fraud detection, and another for billing. All reading the same ride-requested topic. Each operated independently at different speeds, with different processing logic, zero coupling.

You can replay history by resetting offsets

An offset is simply the position of a consumer in a partition's log. Consumer Group A has read up to offset 10,000. Consumer Group B has read up to offset 5,000. Both can move forward, but you can also move backward.

Need to reprocess the last 24 hours because you found a bug? Reset the offset. Need to test a new consumer implementation against historical data? Reset the offset. Need to rebuild a cache or data warehouse? Reset the offset. This capability alone has saved me countless times in production.

// Using kafkajs to reset offsets to an earlier point
const admin = kafka.admin();
await admin.connect();

// Reset 'my-group' to the beginning of the topic
await admin.setOffsets({
  groupId: 'my-group',
  topic: 'user-events',
  partitions: [
    { partition: 0, offset: '0' },  // Start from the beginning
    { partition: 1, offset: '0' },
  ],
});

await admin.disconnect();

// Now when 'my-group' consumers restart, they'll re-read from offset 0

Compare this to a queue. Once you've processed and deleted messages, they're gone. If you need to reprocess, you hope you logged the data somewhere, or you're out of luck.

In a trading platform I worked on, we deployed a calculation engine with a precision bug. By the time we caught it, the consumer had processed hours of market data. We rolled back the service, reset the consumer group's offset to a point before the bug was introduced, and replayed. All historical calculations were corrected. With a queue, we'd have had to manually reconstruct that data or risk stale calculations.

Common mistakes

• Treating Kafka as a task queue. I've seen teams use Kafka to distribute work (like "process this CSV file") and expect messages to disappear. Then they're confused when messages persist and think they have a bug. Use Kafka for events; use a job queue library for distributed tasks.
• Forgetting that offset tracking is per group. New developers sometimes assume that resetting one consumer's offset affects all consumers. It doesn't. Only consumers in that group are affected. This can lead to confusion about which group is "behind."
• Not configuring retention appropriately. Set retention too short and you lose the ability to replay. Set it too long and disk usage becomes expensive. Think through your use cases upfront.
• Assuming message deletion for privacy/compliance. If you need to delete sensitive data from Kafka (like PII), you need special tooling. Retention policies don't delete selectively. This is a real operational challenge in regulated industries.
• Mixing queue and log patterns in the same topic. If you want some consumers to "claim" exclusive ownership of a message (queue-like) and others to replay freely (log-like), you'll end up with coordination problems. Keep your architecture intent clear.

Key takeaways

• Kafka topics are immutable event logs, not ephemeral queues. Messages persist for a configurable retention period.
• Multiple independent consumer groups can read the same topic at different paces, enabling versatile multi-consumer architectures.
• Consumer offsets are tracked per group, allowing you to replay events or catch up without affecting other consumers.
• Offset resets let you reprocess historical data—invaluable for bug fixes, testing, and rebuilding derived state.
• This architecture is powerful but requires a mental shift if you're coming from traditional messaging systems.