Kafka in one picture: a practical Kafka & WebSockets guide

Kafka in one picture

If you're building systems that need to handle millions of events reliably, you've probably heard that Kafka is the answer—but Kafka in one picture isn't a single visual. It's five core concepts that unlock everything else. Once you understand these five ideas, the rest of Kafka stops feeling like magic and starts feeling like a well-designed tool.

I learned Kafka the hard way, shipping it into production on a ride-hailing platform with minimal handover. I want to save you that pain. Let's break down the five concepts that hold Kafka together, then explore why they matter for the systems you're building.

Core Concepts: The Five-Step Flow

Every Kafka system follows the same pattern: Producer writes events → Topic stores them, split into partitions → Broker hosts partitions → Consumer group reads them → Offsets track progress. Let me walk through this with a real example.

Imagine you're building a ride-hailing platform. A driver completes a trip and your backend fires an event: a JSON object with trip ID, driver ID, passenger ID, fare, and timestamp. That's the producer. It sends this event to a Kafka topic called ride-completed.

The topic doesn't store events in a single place. Instead, it splits them across partitions. You might have 4 partitions, and Kafka assigns each event to a partition (usually by key—maybe driver ID). Each partition is an ordered log of events, and each broker (a Kafka server) hosts one or more partitions. So your 4 partitions might be spread across 3 brokers.

Now a consumer wants to process completed rides to calculate driver earnings. It doesn't pull from the entire topic; it joins a consumer group and reads from its assigned partitions. If you have 4 partitions and 2 consumers in the group, each consumer gets 2 partitions. Kafka automatically rebalances if you add or remove consumers.

The offset is Kafka's memory. It tracks which events the consumer has already processed. If your earnings service crashes, it restarts and reads its last offset, then continues from where it left off. No events are lost; no events are processed twice (within reason).

Here's a minimal producer and consumer in TypeScript using the kafkajs library:

import { Kafka } from 'kafkajs';

const kafka = new Kafka({
  clientId: 'ride-app',
  brokers: ['localhost:9092'],
});

// Producer
const producer = kafka.producer();
await producer.connect();

await producer.send({
  topic: 'ride-completed',
  messages: [
    {
      key: 'driver-123',
      value: JSON.stringify({
        tripId: 'trip-999',
        driverId: 'driver-123',
        fare: 2500,
        timestamp: Date.now(),
      }),
    },
  ],
});

// Consumer
const consumer = kafka.consumer({ groupId: 'earnings-service' });
await consumer.connect();
await consumer.subscribe({ topic: 'ride-completed' });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const ride = JSON.parse(message.value.toString());
    console.log(`Processing ride:`, ride);
    // Update driver earnings in your database
    await updateDriverEarnings(ride.driverId, ride.fare);
  },
});

This flow is the skeleton of every Kafka architecture you'll build. Producers and consumers are decoupled. Topics are persistent. Offsets give you fault tolerance. Everything else is a variation or optimization of this pattern.

It's a Log, Not a Queue

This is the mental shift that separates Kafka from traditional message queues like RabbitMQ. In a queue, when a consumer reads a message, it's gone. In Kafka, events are retained (by default for 7 days, but you can configure it). Multiple consumer groups can read the same events independently.

Picture your ride-hailing platform again. You have the ride-completed topic. The earnings service reads every event to calculate driver payouts. But the analytics team also wants to read the same events to build hourly ride reports. And the fraud detection service wants to read them to flag suspicious patterns.

In a queue-based system, you'd have to choose: each service gets one shot at the data, or you duplicate the producer logic. In Kafka, each service joins its own consumer group. The earnings service's group is at offset 5000. The analytics group is at offset 4500 (because it started later). The fraud group is at offset 5200. All three groups read the same topic, at their own pace, tracking their own offsets.

This is powerful. You can add a new consumer group—say, a machine learning pipeline that wants to retrain a model on ride patterns—without touching the producer or other consumers. The ML pipeline starts reading from an earlier offset and catches up over time.

In production, I've seen this pattern save weeks. When the fraud team discovered a new pattern to detect, we didn't need to re-emit historical data or refactor our architecture. We spun up a new consumer group with a fresh topic and let it read through the entire history.

The trade-off: Kafka stores more data and uses more disk space than a traditional queue. If you have 10 consumer groups, Kafka needs to keep events around until the slowest group has processed them. You manage retention with configuration, but you still pay a storage cost.

Partitions Are the Unit of Scale

Kafka's throughput is limited by the number of partitions. More partitions mean more parallel consumers and higher total throughput. This is by design.

Let's say your ride-completed topic has 1 partition and you want to process 100,000 events per second. You're bottlenecked. A single consumer reads from that partition, and no amount of adding more consumers helps—Kafka won't assign the same partition to multiple consumers in the same group because event ordering would break.

The fix: increase the partition count to 4 or 8. Now you can run 4 or 8 consumers in parallel, each processing 25,000 events per second. Your total throughput scales linearly with consumer count (up to the partition count).

But here's the catch: if you're keying events (like by driver ID), the partition key determines which partition each event goes to. All events for a given driver must go to the same partition to preserve ordering. If you have 100,000 drivers and 4 partitions, each partition gets roughly 25,000 drivers. This is usually fine—you're trading global ordering for parallel throughput, and in most systems, you only care about order within a driver's events, not across all drivers.

In the ride-hailing platform, we partitioned by driver ID. This meant the earnings service could run multiple instances, each consuming from different partitions, each updating driver earnings independently. Ordering was preserved per driver, which is what the finance team needed for payout accuracy.

Increasing partitions is easy. Decreasing them is not. Once you set a topic to 8 partitions, changing it back to 4 requires recreating the topic and reprocessing data. So plan ahead. A good heuristic: start with the number of consumer instances you expect to run, then add 20% headroom.

Consumers Pull, Not Push

This is a subtle but critical design choice. Kafka consumers pull events from brokers at their own pace. Brokers don't push events to consumers.

Why does this matter? Control. Your consumer decides when to fetch, how many events to fetch, and when to commit its offset. This means your consumer can adapt to its load. If your database is slow, the consumer fetches fewer events and waits longer between batches. If your database is fast, the consumer fetches larger batches. You control the backpressure.

In contrast, a push-based system (like traditional pub-sub) means the broker decides the pace. If the broker pushes too fast, the consumer is overwhelmed. If the consumer is slow, the broker is blocked or events pile up in memory. This is why push-based systems are harder to scale smoothly.

Here's how it works in practice:

const consumer = kafka.consumer({
  groupId: 'earnings-service',
  sessionTimeout: 30000,
  heartbeatInterval: 3000,
});

await consumer.subscribe({ topic: 'ride-completed' });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const ride = JSON.parse(message.value.toString());
    
    // Your consumer controls the pace here
    try {
      await updateDriverEarnings(ride.driverId, ride.fare);
      // Offset is committed automatically after processing
    } catch (error) {
      // If database is overloaded, this will backpressure
      // The next fetch won't happen until this completes
      console.error('Failed to process:', error);
      throw error; // Requeue or dead-letter
    }
  },
  eachMessageConcurrency: 10, // Control parallelism per consumer
});

The consumer pulls in a loop. It fetches a batch of events, processes them, commits the offset, then fetches the next batch. If processing is slow, the next fetch waits. If processing is fast, the next fetch happens immediately. Your system naturally adapts.

One more thing: heartbeats. While your consumer is processing, it sends periodic heartbeats to the broker to say "I'm alive." If the consumer crashes or hangs for longer than the sessionTimeout, Kafka assumes it's dead and rebalances, reassigning its partitions to other consumers. This is why you see heartbeatInterval and sessionTimeout in production configs—they control how quickly you detect failures.

Common Mistakes

Mistake 1: Not handling offset commits correctly. By default, kafkajs auto-commits offsets, but if your processing fails, the offset is still committed and you lose data. I've seen this in production. The fix: use manual commits or dead-letter topics. Only commit after you're 100% sure the event is processed.

Mistake 2: Not planning for partition count. I've seen teams start with 1 partition, then hit throughput limits and panic when they realize they can't easily increase partitions without losing order or re-creating the topic. Start with more partitions than you think you need.

Mistake 3: Treating Kafka like a database.** Kafka is an event log, not a queryable database. If you need to retrieve "all rides for driver-123 between 10 AM and 11 AM," you need to either query a database or maintain a separate index. Kafka is for streaming and event replay, not random access.

Mistake 4: Ignoring consumer group rebalancing.** When a consumer crashes or joins a group, Kafka rebalances. During rebalancing, consumers stop processing. If you have 4 consumers and 1 crashes, the remaining 3 are paused for several seconds while rebalancing happens. Plan for this latency in your SLAs.

Key Takeaways

• Kafka's five-step flow (producer → topic → partitions → brokers → consumer group → offsets) is the foundation. Understand this and you understand Kafka.
• Kafka is a log, not a queue. Events persist, and multiple consumer groups can read the same events independently. This is your superpower for decoupling services.
• Partitions are your unit of scale. More partitions means more parallel consumers. Plan your partition count upfront; changing it later is painful.
• Consumers pull at their own pace. This gives you natural backpressure and makes your system resilient to variable load.
• Offsets track progress per consumer group. Commit offsets carefully to avoid replaying events or losing data.
• Kafka is not a database. Use it for event streaming and replay. Use a database for random access queries.

That's Kafka in five concepts. In the next post, we'll build on this foundation and talk about WebSockets. See you then.