Replication and in-sync replicas: lessons from production

Kafka brokers fail. Disks corrupt. Networks partition. If your entire event stream lives on a single machine, one bad day wipes out everything. That's where replication and in-sync replicas come in—they're the difference between a hobby project and a system that survives the real world. Understanding how Kafka replicates data across brokers is the foundation of building reliable event-driven systems at scale.

Leaders and followers — the partition hierarchy

Every partition in Kafka has one leader and zero or more followers (we call them replicas). The leader handles all reads and writes for that partition. Followers sit quietly, pulling messages from the leader and keeping an identical copy of the log. When a broker dies, Kafka automatically elects a new leader from the surviving followers, and producers and consumers reconnect to it.

Think of it like this: you're running an e-commerce platform. Order events stream into a topic with three partitions, each with a replication factor of 3. Partition 0's leader is Broker A, with followers on Brokers B and C. Your order service writes to the leader (Broker A), and Brokers B and C pull those messages asynchronously. If Broker A crashes at 3 AM, the cluster automatically promotes one of the followers to be the new leader, and order writes resume on the new leader within seconds.

The replication factor is set at topic creation time and applies to every partition:

// Creating a topic with replication factor 3
// (You'd run this via CLI, but shown here conceptually)
// kafka-topics --create \
//   --topic orders \
//   --partitions 3 \
//   --replication-factor 3 \
//   --bootstrap-servers localhost:9092

// In your Kafka broker config, replication is transparent
// Brokers coordinate leader election automatically
// Your producer just connects to the leader

The beauty of this design is that producers and consumers never need to know about followers. They only talk to the leader. Replication happens behind the scenes, and the cluster handles failover without your application code changing a line.

ISR — In-sync replicas and the lag problem

Here's the trap that catches most teams: just because a follower exists doesn't mean it's caught up. A follower can be slow, network-lagged, or temporarily broken. An in-sync replica (ISR) is a replica that is actively caught up with the leader—it has replicated all messages up to the leader's high watermark.

Kafka tracks the ISR list for each partition. When a follower falls behind (say, it processes messages slower than they arrive, or the network hiccups), it gets removed from the ISR. When it catches up again, it rejoins. You can see this in the broker logs and metrics:

// In your monitoring dashboard, you'd watch:
// kafka.server:type=ReplicaManager,name=UnderReplicatedPartitions
// This metric tells you how many partitions have ISR < replication factor

// If a follower is lagging, Broker logs show something like:
// "Partition orders-0 is not under-replicated but ISR changed from [0,1,2] to [0,1]"
// Broker 2 fell behind and was kicked out of the ISR

Why does this matter? Because if your leader dies and only a lagging follower is available, Kafka has to choose: do we promote a slow replica (risking data loss for messages the dead leader never replicated to it), or do we wait/fail? The answer depends on your configuration, which we'll see next.

A practical scenario: your trading platform receives market data events at high throughput. The leader processes 10,000 events per second. A follower on a slower machine in a distant region can only process 8,000/sec. It falls out of the ISR. If you never configured min.insync.replicas, your producer thinks every write to the leader is "safe"—but if that leader crashes before the slow follower catches up, those 2,000 events/sec are lost forever.

min.insync.replicas — the safety valve

This is where you enforce durability. min.insync.replicas (often abbreviated min.isr) sets the minimum number of in-sync replicas that must acknowledge a write before the leader returns success to the producer. It only matters when you use acks=all (or acks=-1, same thing).

Here's how it works:

1. Producer sends a message with acks=all
2. Leader receives it and appends to its local log
3. Leader checks: how many replicas are currently in the ISR?
4. If fewer than min.insync.replicas are in the ISR, the leader refuses the write and returns an error
5. If enough are in the ISR, the leader waits for them to replicate the message
6. Once min.insync.replicas have acknowledged, the leader returns success to the producer

Let's see this in code:

import { Kafka, logLevel } from "kafkajs";

const kafka = new Kafka({
  clientId: "trading-producer",
  brokers: ["localhost:9092"],
});

const producer = kafka.producer();

// Scenario: Trading platform, orders topic
// Replication factor = 3, min.insync.replicas = 2
await producer.send({
  topic: "orders",
  messages: [
    {
      key: "user-123",
      value: JSON.stringify({
        symbol: "AAPL",
        quantity: 100,
        price: 150.25,
        timestamp: Date.now(),
      }),
    },
  ],
  timeout: 5000,
  compression: 1, // Gzip
  acks: -1, // acks=all: wait for all ISR to acknowledge
});

// What happens behind the scenes:
// 1. Message goes to leader (say, Broker 0)
// 2. Broker 0 checks ISR count for orders partition
// 3. If ISR = [0, 1, 2] (all 3 replicas in sync)
//    → Message replicates to Brokers 1 and 2
//    → Both acknowledge
//    → Leader returns success
// 4. If ISR = [0, 1] (Broker 2 lagged out)
//    → Message replicates only to Broker 1 (1 follower)
//    → That's still 2 in-sync replicas total (leader + 1 follower)
//    → min.insync.replicas = 2 is met
//    → Leader returns success
// 5. If ISR = [0] (Brokers 1 and 2 both lagged/down)
//    → Only the leader is in sync
//    → min.insync.replicas = 2 is NOT met
//    → Producer gets an error: NotEnoughReplicas
//    → The write is rejected

That last scenario is the key safety feature. Your producer catches the error and can decide: retry, alert, or fail gracefully. You never silently lose data.

The trade-off is latency. With min.insync.replicas=2 and a slow follower, writes block until both replicas acknowledge. In high-throughput systems, this can add 10–50ms per write. That's why you tune min.insync.replicas based on your SLA: if you need ultra-low latency, you might accept min.insync.replicas=1. If you run a bank, you want min.insync.replicas = replication_factor - 1.

The safe trio — replication factor 3, min.insync.replicas 2, acks=all

This is the configuration most teams converge on for critical topics. Here's why it's the sweet spot:

• Replication factor 3: You can lose 1 broker and still have quorum. You can tolerate 1 broker maintenance window, 1 network partition, 1 disk failure.
• min.insync.replicas 2: You need at least 2 copies of every message (the leader + 1 follower) before the write succeeds. If the leader dies after the write returns, the surviving followers have the message.
• acks=all: Producer waits for those 2 copies to be written. No "fire and forget" silently losing data.

Applied to our e-commerce scenario:

// Topic config (set at creation or via AdminClient)
const admin = kafka.admin();
await admin.createTopic({
  topic: "order-events",
  numPartitions: 3,
  replicationFactor: 3,
  configEntries: [
    {
      name: "min.insync.replicas",
      value: "2",
    },
  ],
});

// Producer config
await producer.send({
  topic: "order-events",
  messages: [{ key: "order-456", value: orderPayload }],
  acks: -1, // acks=all
  timeout: 5000,
});

// Failure scenario 1: Broker A (leader) dies
// → ISR was [A, B, C], becomes [B, C]
// → New leader elected (say B)
// → min.insync.replicas = 2 is still met (B and C)
// → Writes resume immediately
// → Zero data loss because every message was replicated to B and C

// Failure scenario 2: Broker A dies AND Broker B is slow
// → ISR drops to [A, C], then [C] as A dies
// → min.insync.replicas = 2 is NOT met
// → Producer gets NotEnoughReplicas error
// → You detect the problem and alert ops
// → No silent data loss

This config assumes your topic is truly critical. For less critical data (analytics events, metrics, logs), you might use replication factor 2 or min.insync.replicas 1 to save throughput and storage.

Common mistakes

Setting min.insync.replicas = replication factor: If you do this, losing even 1 broker stops all writes. min.insync.replicas should almost always be replication_factor - 1.

Forgetting to set min.insync.replicas: Many teams create a topic with replication factor 3 but never set min.insync.replicas, leaving it at the default 1. That means the leader alone is enough for acks=all to return success. One dead broker, one lost message.

Confusing ISR with "all replicas": The ISR list is dynamic. It shrinks when replicas lag, and grows when they catch up. Monitoring only the replication factor (static) and ignoring ISR health (dynamic) means you won't see degradation until writes start failing.

Tuning for latency without measuring: Yes, acks=all + min.insync.replicas=2 adds latency. But some teams disable it without measuring what they're saving (10ms?) versus what they're risking (data loss in one failure). Always profile before and after.

Key takeaways

• Every partition has one leader (handles reads/writes) and followers that replicate the log asynchronously.
• In-sync replicas (ISR) are followers currently caught up with the leader; the ISR list shrinks if a follower lags or fails.
• min.insync.replicas with acks=all forces the producer to wait until at least that many replicas have the message before returning success, preventing silent data loss.
• Replication factor 3 + min.insync.replicas 2 + acks=all is the industry standard for critical topics; it tolerates 1 broker failure with zero data loss and minimal latency penalty.
• Always set min.insync.replicas explicitly on critical topics; don't rely on defaults.
• Monitor UnderReplicatedPartitions and ISR changes in your broker metrics; they're early warning signs of cluster problems.