How many partitions do you need: lessons from production

How many partitions do you need?

Partition count is one of those Kafka decisions that feels simple until it isn't. Pick too few and you'll throttle throughput no matter how many consumers you spin up. Pick too many and you're burning CPU, memory, and network bandwidth on coordination overhead that doesn't buy you anything. Getting this right means understanding your actual throughput requirements, planning for growth, and matching your partition count to how many consumers can realistically process in parallel.

Start with expected peak throughput per consumer

The foundation of partition sizing is knowing how much data a single consumer can handle. This isn't theoretical—you need to measure it under realistic conditions.

Let's say you're building an order processing system. You run a test where a single consumer processes orders from one partition and measure end-to-end latency and throughput. You find that with your current business logic—validation, database writes, maybe an HTTP call to a payment service—one consumer can reliably handle 500 orders per second before latency creeps above acceptable limits.

Now you estimate peak load. Black Friday might hit you with 50,000 orders per second across all topics. Simple division: 50,000 / 500 = 100 consumers needed to keep up. Since each consumer needs its own partition to avoid coordination bottlenecks, you need at least 100 partitions.

But this measurement matters more than the math. I've seen teams assume a consumer can handle 1,000 messages per second based on nothing, then wonder why their cluster is melting during peak hours. Run benchmarks. Use a tool like JMeter or write a test consumer that mimics your actual workload—network calls, database queries, all of it. The number you get is your throughput baseline.

// Example: measure consumer throughput
const startTime = Date.now();
let messagesProcessed = 0;

consumer.on('message', async (message) => {
  try {
    // Your actual business logic
    await validateOrder(message.value);
    await database.orders.insert(message.value);
    await paymentService.charge(message.value.payment);
    
    messagesProcessed++;
  } catch (error) {
    console.error('Processing failed:', error);
  }
  
  const elapsed = (Date.now() - startTime) / 1000;
  if (elapsed > 60) {
    const throughput = messagesProcessed / elapsed;
    console.log(`Throughput: ${throughput.toFixed(2)} msg/sec`);
    process.exit(0);
  }
});

Once you have your baseline, multiply it by a safety factor—usually 1.5 to 2. This gives you headroom for garbage collection pauses, database slowdowns, or unexpected spikes. If one consumer can handle 500 msg/sec, assume 250–300 for planning purposes.

Plan for growth—increasing later reshuffles keys

Here's the painful part: you can't easily increase partition count without reshuffling data. Kafka doesn't rebalance existing messages across new partitions. If you add partitions later, new messages follow the new partition scheme, but old messages stay where they are. For stateless consumers this is fine, but for anything that depends on ordering or key-based grouping, it's a mess.

Consider a user activity topic where key is user ID. Messages for a given user always go to the same partition, so all their events arrive in order. If you start with 10 partitions and later jump to 50, the hash function that maps keys to partitions changes. User 12345 that was in partition 3 is now in partition 17. New events arrive in the new partition, but their historical data is scattered. Any consumer trying to replay or aggregate by user now has to read from multiple partitions and manually sort. Painful.

The workaround is to do a full topic migration—write a tool that reads from the old topic and republishes to a new topic with more partitions. It works, but it's downtime or at least a significant operational effort.

So plan for 2–3 years of growth from day one. If you expect linear growth and hit peak load in year 2, size for 1.5–2x that peak. It's cheaper to have some unused partition overhead than to redo your entire topic schema.

Also consider your retention policy. A high-throughput topic retained for 30 days uses far more disk than one retained for 1 day. The partition count affects how data is distributed across brokers, which affects storage and network load.

Dozens per topic is common; thousands needs care

In most production setups I've worked with, topics have 20–200 partitions. A typical e-commerce order topic might have 48 partitions (4 brokers × 12 partitions per broker for parallelism). A high-volume event stream could go up to 256. These numbers work well because they're large enough for scale but small enough that broker coordination, controller overhead, and rebalancing times stay reasonable.

But I've seen teams think "more partitions = more scale" and jump to 1,000 or 5,000 partitions on a single topic. At that scale, you start hitting real problems.

Controller elections take longer. Metadata replication costs more. Each broker maintains state for every partition on every topic, so thousands of partitions × dozens of brokers = millions of small state objects. Rebalancing becomes glacially slow—a new consumer joining a consumer group with 2,000 partitions might take minutes to stabilize. ZooKeeper (or KRaft) can become a bottleneck.

If you genuinely need thousands of partitions, consider splitting into multiple topics. Instead of one events topic with 2,000 partitions, use events-region-us, events-region-eu, events-region-asia with 500 partitions each. You gain operational clarity and avoid the coordination tax of a single massive topic.

Match partitions to max consumer parallelism

Here's a principle I always follow: the maximum useful number of partitions equals the maximum number of consumers you'll ever run for that topic. There's no benefit to having more partitions than consumers.

If you have 48 partitions but only 10 consumers, 38 partitions sit idle. The 10 consumers each grab 4–5 partitions and that's fine, but you're not gaining anything from the extra 38. They still consume memory on brokers, still get replicated, still show up in metadata. Conversely, if you have 10 partitions and 48 consumers, 38 consumers are idle. Only 10 can do useful work. The rest are deadweight in the consumer group, burning resources while waiting for a rebalance that might never give them a partition.

So think about your deployment model. For an order processing system, you might run 12 replicas of your consumer service. If orders can be processed in any order, 12 partitions is ideal. If you scale up to 24 replicas, bump to 24 partitions. The sizing stays proportional.

// Example: calculate ideal partition count
const estimatePeakThroughput = 50_000; // msg/sec
const singleConsumerThroughput = 500; // msg/sec (from benchmarking)
const safetyFactor = 1.5;

const minConsumersNeeded = Math.ceil(
  estimatePeakThroughput / (singleConsumerThroughput * safetyFactor)
);

// Also consider max replicas you might ever deploy
const maxReplicas = 50;

const recommendedPartitions = Math.max(minConsumersNeeded, maxReplicas);

console.log(`Recommended partitions: ${recommendedPartitions}`);
// Output: ~83 (based on the numbers above)

Plan for the maximum reasonable deployment. If your system might scale to 50 instances, use 50–100 partitions. Unused partitions cost very little; unused consumer parallelism costs a lot.

Common mistakes

• Starting with too few partitions. Teams often begin with 3 or 5 partitions "to keep it simple" and assume they can scale later. Reshuffling partitions is a migration. Start bigger; you won't regret it.
• Sizing based on message size, not message count. A partition with huge messages (10 KB each) and one with tiny messages (100 bytes) need different partition counts based on message rate, not payload.
• Ignoring broker disk distribution. If you have 3 brokers and 97 partitions, the distribution is uneven (some brokers get 33, others get 32). Uneven partition distribution causes hot brokers. Use multiples of your broker count, or accept the imbalance.
• Not accounting for replicas in sizing. Partition count affects both leaders and replicas. 100 partitions with replication factor 3 means 300 partition objects cluster-wide. Monitor broker memory.
• Creating separate topics when partitions would do. The opposite mistake: over-partitioning a single topic instead of splitting into multiple topics. If you're approaching 500 partitions, reconsider your topic design.

Key takeaways

• Measure your single-consumer throughput under realistic load, then divide peak throughput by that number (with safety margin) to find minimum partition count.
• Increasing partitions later requires a full topic migration. Size for 2–3 years of growth upfront.
• Most production topics live in the 20–200 partition range. Thousands of partitions add real operational overhead.
• Match your partition count to your maximum realistic consumer parallelism. More partitions than consumers is waste; fewer partitions is a bottleneck.
• Prototype and benchmark before choosing a final number. The cost of getting it wrong is high enough that 2–3 hours of testing is worth it.