Partitions and ordering: real-world lessons and examples

You've deployed your first Kafka topic and started sending events. Everything works — until your ride-hailing app loses the order of driver location updates for a single ride, or your e-commerce pipeline processes refunds before charges. The culprit isn't Kafka's fault; it's a misunderstanding of how ordering actually works. Kafka guarantees order — but only within a partition, and which partition an event lands in depends entirely on the key you choose. This article walks through the real-world lessons and examples that will save you hours of debugging.

Order per partition only: the fundamental guarantee

Kafka's ordering promise is narrower than many assume. When you send messages to a topic, Kafka guarantees that messages within the same partition are processed in the order they arrive. But if your topic has three partitions and you send ten messages without a key, those messages might be distributed across all three partitions, and consumers could see them in any order.

Think of it like a post office with three mail sorters. Each sorter processes letters in the order they receive them, but the post office can hand-off incoming mail to any sorter. If you care that letter #5 is opened before letter #10, you need to ensure both go to the same sorter.

In production systems handling financial transactions, user state, or time-sensitive events, losing order causes real damage. A payment system that processes a refund before the original charge, an inventory system that decrements stock before confirming payment, or a chat app that shows a reply before the original message — all of these come from misunderstanding partition-level ordering.

The architecture of Kafka enforces this constraint by design. Each partition is maintained on a single broker (or replicated across a few), and a single consumer in a consumer group can only read from one partition at a time. This guarantee is powerful, but you must be intentional about which events land in which partition.

Keys decide partitions: routing messages to consistency

The partition a message lands in is determined by the key you provide when sending. If you don't provide a key, Kafka uses round-robin distribution, scattering your messages across partitions. If you do provide a key, Kafka runs it through a hash function (by default, a murmur2 hash) to map it consistently to a partition.

The critical insight: the same key always hashes to the same partition. This means all events with the same key will be processed in order. Here's how it works in a ride-hailing context:

const producer = kafka.producer();
await producer.connect();

const rideId = '550e8400-e29b-41d4-a716-446655440000';
const event = {
  type: 'location_update',
  timestamp: Date.now(),
  latitude: 37.7749,
  longitude: -122.4194,
};

await producer.send({
  topic: 'rides',
  messages: [
    {
      key: rideId,  // Same rideId always goes to the same partition
      value: JSON.stringify(event),
    },
  ],
});

In this example, every location update for ride 550e8400-e29b-41d4-a716-446655440000 will go to the same partition (let's say partition 2). The driver's position updates will be processed in order: first at latitude 37.7, then 37.77, then 37.775. A consumer reading from partition 2 will never see the positions out of sequence.

Without the key, location updates for the same ride could land in partitions 0, 1, or 2 randomly. A consumer might process the 37.775 update before the 37.7 update, resulting in a map showing the driver moving backward — or worse, a system state where the ride appears complete before it actually started.

Choose keys by entity: aligning business logic with partitioning

The art of using Kafka well is choosing the right key for your domain. The key should be whatever entity needs to maintain consistency. For an e-commerce system, that's typically the orderId. For a ride-hailing app, it's the rideId. For a messaging platform, it might be the conversationId or userId depending on your guarantees.

Consider an order lifecycle in a SaaS platform:

All events use orderId as the key. They all hash to partition 1. A consumer processing partition 1 will never see shipment_created before payment_authorized. Your downstream systems can build accurate state machines.

This doesn't mean you can't have other fields in your event payload. You might include userId, status, amount, or region for filtering, analytics, or enrichment. But the key is specifically about routing and ordering, so choose it with that in mind. If you need to guarantee ordering across a user's entire session, use userId. If you need per-order consistency, use orderId. You can't have both with a single key—that's a fundamental trade-off in distributed systems.

Hot partitions: when one key overloads your infrastructure

Choosing the right key is not just about correctness; it's about operational sustainability. If a single key receives vastly more traffic than others, that key's partition becomes a bottleneck—a hot partition. While other partitions are idle, one partition is saturated, and the broker hosting it becomes a performance cliff.

Real-world example: a ride-hailing platform handles driver location updates. Most drivers send updates every 5 seconds. But your highest-paying premium customer is a corporate shuttle service with 50 vehicles. They have an internal ID that you're using as the key. All 50 vehicles hash to partition 7. Partition 7 now receives 10× more throughput than any other partition, while partition 2 sits half-empty. Your single partition can only handle so much, and you've just throttled yourself artificially.

Detecting hot partitions requires monitoring:

• Monitor bytes-in and record-in per partition at the broker level.
• Look for one partition with significantly higher throughput than others.
• Track consumer lag per partition; a hot partition will have higher lag because the consumer can't keep up.
• Use Kafka metrics like kafka.server:type=ReplicaManager,name=InSyncReplicasCount to watch replica health under load.

When you detect a hot partition, you have a few options:

1. Re-partition the key: Instead of using a single customerId, use customerId + regionId or customerId + vehicleNumber. This distributes the load, but you lose ordering across the entire customer's events. This is a real trade-off.
2. Scale the topic: Increase the partition count and rebalance. This spreads load, but existing consumers must rebalance. For streaming applications that maintain state, this can be painful.
3. Accept the hot partition and add more brokers: If partition 7 is hot but still performant, and you have headroom on that broker, sometimes the simplest answer is to buy more capacity.
4. Decouple into multiple topics: If different event types have different ordering requirements, split them. Premium customer location updates in one topic keyed by vehicle, standard customer updates in another keyed by driver. Process them with different parallelism.

The key lesson: choose your key based on business requirements (what consistency do you need?), not just technical convenience. Then monitor partition skew as part of your operational routine.

Common mistakes

• Sending messages without a key and expecting order: If you don't provide a key, messages are round-robined across partitions. Order is not guaranteed. Always set a key if ordering matters.
• Using a high-cardinality key that varies per message: Using a timestamp, UUID, or request ID as a key ensures every message goes to a different partition. This kills ordering entirely and also creates hot partitions if the hash is biased.
• Assuming Kafka order applies across topics: Ordering is per partition, per topic. Events in orders and payments topics have no guaranteed relative order, even if sent by the same producer with related keys.
• Forgetting to handle rebalancing: When consumers rebalance (due to scaling or failures), the partition-to-consumer assignment changes. A consumer that was reading partition 1 might now read partition 3. If your consumer maintains local state, rebalancing can cause duplicate processing or state loss. Use exactly-once semantics or idempotent processing.
• Ignoring partition count when designing for scale: A topic with 3 partitions can only be consumed by a maximum of 3 consumers in a group. If you need 10 concurrent consumers, you need at least 10 partitions. This is easy to overlook in early-stage development.

Key takeaways

• Kafka guarantees order within a partition only. Messages in different partitions have no ordering guarantee.
• The key you provide when sending determines which partition a message lands in. The same key always hashes to the same partition.
• Choose your key based on the entity that needs consistency: orderId for orders, rideId for rides, userId for user events.
• Monitor partition skew and throughput distribution. A hot partition with uneven load can become a bottleneck.
• Be intentional about trade-offs: more parallelism (more keys, more partitions) means losing global ordering. Tighter ordering requirements mean lower throughput per partition.
• Test ordering guarantees explicitly. Write a consumer that validates event sequence and alerts if it detects out-of-order processing.