Running Kafka in production: what developers should know

Running Kafka in production: what developers should know

You've tested Kafka locally, validated your message flow, and now you're preparing for production. The jump from a laptop cluster to handling real traffic is where theory meets painful reality—and most teams learn the hard way that storage fills up, brokers fall over during upgrades, and message retention becomes a heated conversation in standup. Running Kafka in production: what developers should know covers the operational decisions that separate a stable pipeline from a crisis on Friday night.

This article dives into sizing, retention strategy, upgrade procedures, and the managed-versus-self-hosted decision. You'll walk away with a practical mental model for production Kafka, informed by teams that have scaled from thousands to millions of messages per second.

Size for peak, plus headroom

Kafka sizing requires three inputs: your target throughput, message retention period, and replication factor. The formula is straightforward in concept but often underestimated in practice:

// Disk space required = (throughput_MB_per_second * retention_seconds * replication_factor) + 20% headroom

// Example:
// - 100 MB/s peak throughput (this is your 99th percentile burst, not average)
// - 7 days retention
// - Replication factor 3

const throughputMBps = 100;
const retentionDays = 7;
const retentionSeconds = retentionDays * 24 * 60 * 60; // 604,800 seconds
const replicationFactor = 3;
const headroomPercent = 1.2; // 20% headroom

const diskGB = (throughputMBps * retentionSeconds * replicationFactor * headroomPercent) / 1024;
console.log(`Required disk per broker: ${diskGB.toFixed(0)} GB`); 
// Output: Required disk per broker: 21,862 GB (~22 TB per broker)

Three critical points often missed:

• Peak, not average. Monitor your 99th percentile throughput, not your average. A topic averaging 10 MB/s can burst to 50 MB/s during peak hours or batch events. Size for the burst.
• Replication factor is multiplicative. A replication factor of 3 means you pay for three copies of every message. This is not optional in production—it's your insurance policy—but it's expensive. Some teams run RF=3 for critical topics and RF=2 for less critical ones.
• Headroom prevents cascading failures. When a broker runs out of disk, it goes offline. When a broker goes offline and your replication factor is tight, remaining brokers fill up faster, causing a cascade. Always keep 15–20% free space as a buffer.

Network bandwidth is the second constraint. Kafka replicates data across brokers and sends it to consumers. A simple rule of thumb: network bandwidth needed equals (throughput * (replication_factor + number_of_consumer_groups)). If you have 3 replicas and 5 active consumer groups reading the same topic, that's 8x the baseline throughput flowing across your network.

In practice, this means:

• A 100 MB/s topic with RF=3 and 5 consumer groups needs ~800 MB/s network capacity just for that topic.
• Don't assume 10 GbE is enough for "medium" clusters. Network becomes the bottleneck faster than you'd expect.
• Use tiered storage (log compaction, time-based segment retention) to keep hot data small and cold data separated.

Retention by use case

Retention policy determines how long Kafka keeps messages before deletion. It's a business and operational decision, not a technical default, and it should vary by topic.

Operational topics (real-time processing, order events, user actions): Keep 1–3 days. These topics feed real-time services that consume immediately. If a consumer falls behind by more than a day or two, replaying older data rarely adds value, and you've probably already handled the event in a data warehouse. Three days is a safe middle ground that lets you recover from a consumer downtime over a weekend.

Replayable history (audit logs, compliance events, financial transactions): Keep 30–90 days or longer. These topics support debugging, regulatory requirements, and occasional deep reanalysis. The cost of storage is acceptable because these topics are usually narrower in throughput, and losing history is unacceptable.

State reconstruction topics (user profiles, inventory snapshots): Keep as long as your business model allows, ideally with log compaction enabled. Log compaction keeps only the latest value per key, so storage grows logarithmically, not linearly. This lets new instances or recovered consumers rebuild state quickly without replaying the entire history.

Here's how to set retention in your topic configuration:

// Create a topic with time-based retention (3 days)
kafka-topics.sh --create \
  --topic orders \
  --partitions 12 \
  --replication-factor 3 \
  --config retention.ms=259200000 \  // 3 days in milliseconds
  --config min.insync.replicas=2

// Or alter an existing topic
kafka-configs.sh --bootstrap-server localhost:9092 \
  --entity-type topics \
  --entity-name audit-logs \
  --alter \
  --add-config retention.ms=7776000000  // 90 days

// For compacted topics (keep latest value per key)
kafka-topics.sh --create \
  --topic user-profiles \
  --partitions 6 \
  --replication-factor 3 \
  --config cleanup.policy=compact \
  --config min.cleanable.dirty.ratio=0.5 \
  --config delete.retention.ms=86400000  // Keep deletes for 1 day

A common mistake: setting retention too aggressively to save space. If you're tight on disk, the solution isn't shorter retention—it's bigger brokers or fewer replicas (less safe) or better topic design. Short retention (12 hours) often leads to surprised teams when a consumer process crashes and can't rewind far enough to recover.

Rolling upgrades

Kafka cluster upgrades are one of the most high-stakes operational tasks. Stop all brokers at once, and you lose the cluster. But with proper replication and a careful rolling strategy, you can upgrade brokers one at a time with zero downtime.

The rolling upgrade process:

1. Set the cluster-wide minimum in-sync replicas (ISR) to 2: min.insync.replicas=2. This ensures a partition is writable only if at least 2 replicas are in sync, protecting against data loss during the upgrade.
2. Stop one broker.
3. Upgrade the broker software and configuration.
4. Start the broker and wait for it to rejoin the ISR and catch up on replication lag.
5. Monitor the broker's recovery metrics; once caught up, move to the next broker.
6. Repeat until all brokers are upgraded.

Monitoring during a rolling upgrade is critical. Watch these metrics:

// Key metrics to monitor during rolling upgrades:

// 1. Under-replicated partitions (should be 0 or very low)
// Query via JMX or Prometheus:
kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions

// 2. ISR shrinkage and expansion (should stabilize after broker rejoins)
kafka.server:type=ReplicaManager,name=IsrShrinksPerSec
kafka.server:type=ReplicaManager,name=IsrExpandsPerSec

// 3. Replication lag (bytes behind leader)
kafka.server:type=ReplicaFetcher,name=BytesBehindLeadingReplica,clientId=Replica

// 4. Broker startup time (watch logs for "Started up 1 replica fetcher" messages)

// Practical check: verify ISR health before moving to the next broker
kafka-topics.sh --bootstrap-server localhost:9092 --describe | grep -E "Isr|Leader"
// All partitions should show Isr with >= 2 replicas before proceeding

Pro tip: don't upgrade during peak traffic or just before a major campaign. Schedule upgrades during a maintenance window or low-traffic period. If upgrading a major version (e.g., 2.8 to 3.0), test thoroughly in staging first—protocol changes, configuration defaults, and performance tuning can differ significantly.

Managed unless you must

Running self-hosted Kafka means managing disk failures, network partitions, broker crashes, certificate rotations, schema registry uptime, monitoring, alerting, backups, and disaster recovery. It's not just deploying containers—it's being on-call for Kafka.

For most teams, managed services remove significant operational burden:

• AWS Managed Streaming for Kafka (MSK): AWS handles broker provisioning, patching, monitoring, and failover. You focus on topics and consumers. Cost is higher than self-hosted, but the operational overhead is dramatically lower. MSK integrates with CloudWatch, VPC, IAM, and Prometheus.
• Confluent Cloud: Fully managed, multi-cloud Kafka with Schema Registry, Connect, and ksqlDB included. Best if you're already in the Confluent ecosystem or need advanced features out of the box.
• Self-hosted on Kubernetes: Gives you control and can be cost-effective at scale, but requires expertise in Kafka operations, Kubernetes networking, persistent storage, and disaster recovery.

Decision framework:

If you choose self-hosted, you'll want to invest in:

• A robust monitoring stack (Prometheus + Grafana or Datadog).
• Automated alerting for disk usage, replication lag, broker downtime, and consumer lag.
• A documented runbook for common failures: broker crashes, disk full, replication lag, network partitions.
• Regular disaster recovery tests (full cluster rebuild from backups).

Common mistakes

Underestimating peak throughput. Teams size Kafka for average throughput, then get surprised when a marketing campaign or batch job causes a spike. Always use the 99th percentile, plus 50% headroom. It's cheaper than emergency scaling at 2 AM.

Setting replication factor to 1 to save space. It might feel wasteful, but RF=1 means a single broker failure loses data. Production Kafka should always have RF ≥ 2, ideally 3. The storage cost is worth the insurance.

Not monitoring replication lag. Under-replicated partitions are Kafka's silent killer. Brokers can be "up" but not catching up with replication, leaving your cluster vulnerable. If a leader fails while replicas are behind, data loss is possible.

Ignoring retention until disk is full. Teams run Kafka without a clear retention strategy, and eventually, disk fills up overnight. Plan retention upfront as part of topic design, not as an afterthought.

Upgrading during peak hours. Rolling upgrades are safe in theory but can cause latency spikes or consumer lag during the rebalancing. Always schedule during maintenance windows.

Key takeaways

• Size Kafka for your 99th percentile peak throughput, not your average, and include a 15–20% disk buffer to prevent cascading failures.
• Retention is a business decision: 1–3 days for operational topics, 30+ days for replayable or compliance-critical data. Use log compaction for state-heavy topics.
• Rolling upgrades are possible only because of replication; set min.insync.replicas=2, upgrade one broker at a time, and monitor under-replicated partitions closely.
• Managed Kafka (MSK, Confluent Cloud) is the right choice for most teams unless you have deep Kafka expertise and significant multitenancy needs.
• Invest in monitoring replication lag, disk usage, and consumer lag. A good runbook and alerting stack are worth far more than the cost of extra brokers.