Choosing a partition key: a hands-on guide for engineers

The foundation of Kafka's ordering guarantees

Choosing a partition key is the most important Kafka design decision you'll make, yet teams often get it wrong. The partition key determines which broker stores your message and, critically, which messages are processed in order. Get this wrong and you'll face ordering violations, data inconsistencies, or operational nightmares down the line. In this guide, we'll walk through choosing a partition key: a hands-on guide for engineers who need to ship reliable systems quickly.

Why the partition key matters more than you think

Kafka gives you ordering guarantees per partition, not per topic. That means if you send ten messages with the same key, they land on the same partition and are processed in exactly the order you sent them. But if you pick the wrong key, you might inadvertently scatter related messages across partitions, breaking the ordering contract your downstream consumers depend on.

Before we dive into the rules, let's establish what we're trying to solve. A partition key answers one question: which messages must be processed in order together? The answer should reflect your business domain, not convenience.

Key by the entity whose order matters

The golden rule: partition by the entity that owns the state you're tracking. If you're processing financial transactions, the entity is the account. If you're tracking user activity, it's the user. If you're orchestrating an order workflow, it's the order.

Let's look at a concrete example. Imagine you're building an e-commerce platform with order events:

interface OrderEvent {
  orderId: string;
  userId: string;
  status: 'created' | 'paid' | 'shipped' | 'delivered';
  amount: number;
  timestamp: number;
}

// Correct: partition by order
const key = event.orderId;

// Wrong: partition by user (different orders could be interleaved)
const key = event.userId;

// Wrong: partition by status (orders with same status scattered randomly)
const key = event.status;

Why is orderId correct? Because the order is the entity whose state matters. You need to process "created" before "paid" before "shipped". If order messages scatter across partitions, a slow consumer might process "shipped" before "paid" arrives, leading to inconsistent state.

Another example from ride-hailing: if you're tracking ride progression (requested → accepted → en route → completed), partition by rideId, not by driver or passenger. The ride is the entity with the state machine.

The partition key should correspond to the entity whose lifecycle or state transitions matter to your business logic.

High cardinality spreads load evenly

Once you've identified the right entity, ensure it has high cardinality. Cardinality means the number of distinct values. If your key has low cardinality, some partitions will be hot (receive most messages) while others sit idle.

Consider a SaaS platform tracking subscription events:

interface SubscriptionEvent {
  subscriptionId: string;
  accountId: string;
  tenantId: string;
  eventType: 'created' | 'upgraded' | 'cancelled';
  timestamp: number;
}

// High cardinality: ~1M unique subscriptions over time
const key = event.subscriptionId;

// Low cardinality: maybe 50 accounts
const key = event.accountId;

// Even lower: maybe 5 tenants
const key = event.tenantId;

If you partition by accountId and you have a whale customer with 40% of all subscriptions, that account's messages will all hash to one or two partitions while the others sit idle. Your brokers can't parallelize, and you hit throughput limits.

By partitioning by subscriptionId, you distribute load evenly across your broker cluster. Each subscription's events stay ordered, but different subscriptions parallelize naturally.

A practical check: estimate the cardinality of your key over a 24-hour window and divide it by your target partition count. If subscriptions have 1M cardinality and you want 10 partitions, each partition gets ~100K keys on average. That's reasonable. If you only have 10 distinct keys for 10 partitions, you've likely chosen wrong.

Avoid keys like country or status

Low-cardinality fields tempt engineers because they seem like natural groupings. Resist this temptation. Country, status, region, and type are red flags.

Let's say you're building an event system for a travel platform and consider this design:

interface TravelEvent {
  bookingId: string;
  country: string;
  status: 'pending' | 'confirmed' | 'cancelled';
  timestamp: number;
}

// Tempting but wrong: "All bookings in the US should be together"
const key = event.country;

// Also tempting but wrong: "Process all pending events together"
const key = event.status;

The problem: if 60% of your traffic is US bookings and you partition by country, the US partition becomes a bottleneck. Worse, a booking's status changes over time. If you partition by status, a single booking's events scatter across multiple partitions: the "pending" partition gets the first event, the "confirmed" partition gets the second. You've destroyed ordering.

Here's the right approach:

// Correct: partition by booking (the entity with state)
const key = event.bookingId;


Your consumers can still filter or group by country or status in application logic. Kafka should enforce ordering for the business entity, not for reporting dimensions.

Changing keys later breaks ordering guarantees

Many teams pick a partition key, ship it, and later realize they chose wrong. Changing the key is not a simple migration—it's an architecture shift that breaks your ordering guarantees.

Here's what happens when you change keys:


Old messages land on partitions based on the old key.
New messages land on partitions based on the new key.
Your consumers now see interleaved messages from both schemes.
Order is no longer guaranteed for any entity.


You might think "we'll just re-consume from the beginning with the new key," but that's expensive, risky, and requires downtime. Instead, plan your key carefully upfront.

To future-proof yourself, ask these questions before shipping:


What entity's lifecycle are we tracking?
Does this entity's cardinality give us good partition distribution?
If requirements change in 18 months, can we work with this key, or will we be forced to migrate?


If you're unsure, err on the side of a unique identifier over a business property. A UUID or internal ID is more stable than country codes or status enums, which often change.

Common mistakes


Null or missing keys: Kafka treats null keys as "no ordering required" and distributes them round-robin across partitions. If your key is ever null, you've lost ordering for that message. Always ensure your keys are present and stable.
Composite keys without thought: Combining two fields (e.g., userId:orderId as a string) can work, but only if the combined value remains stable. If either component changes, ordering breaks.
Picking keys for consumer convenience: "We want all payments for a region together" tempts you to partition by region. Resist. Your consumer code can filter; Kafka should enforce domain ordering.
Assuming high cardinality without verification: "Order IDs are unique" doesn't guarantee distribution if your traffic is skewed. If 90% of orders are from one customer that day, you still get hotspots.
Changing keys in schema evolution: If you refactor your event schema, ensure the key field's semantics don't change. Renaming a field is fine; changing what it represents is not.


Key takeaways


Partition by the entity whose state or lifecycle you're tracking, not by reporting dimensions.
Verify that your key has sufficient cardinality to spread load evenly across your partition count.
Avoid low-cardinality fields like country, status, or region as partition keys; use them in application logic instead.
Plan your partition key carefully before shipping; changing it later requires a costly re-architecture.
Always ensure your key is non-null and stable for the lifetime of the message.
When in doubt, use a unique identifier (UUID, order ID, booking ID) rather than a business attribute.