Enable the idempotent producer: best practices explained

Retry logic is a double-edged sword in distributed systems: it keeps your application resilient, but it can accidentally turn a single message into multiple copies in Kafka. On Day 11, we'll address this head-on by learning how to enable the idempotent producer, a feature that ensures your retries remain safe and your data stays consistent.

Why duplicates happen and why they matter

When a Kafka producer sends a message, the network or broker might fail after the write succeeds but before the acknowledgment reaches the client. The producer doesn't know if the message landed, so it retries. Without idempotency, that single message is now in Kafka twice, and your downstream consumers might process the same event twice—charging a customer twice, creating duplicate records, or corrupting aggregations.

I've seen this in production ride-hailing platforms where a driver's location update was retried three times due to a transient network hiccup. The consumer interpreted it as three separate movements, throwing off real-time location tracking. That's when idempotent producers become essential.

The cost of duplicates is almost always higher than the cost of preventing them. That's why Kafka's idempotent producer is worth understanding and enabling today.

Enable the idempotent producer

Enabling the idempotent producer is straightforward: set the configuration property enable.idempotence=true on your producer. This single flag tells Kafka's producer to assign a unique sequence number to each message within a partition, and the broker will reject any duplicate within the same producer session.

Here's how you enable it in a Node.js / NestJS application using KafkaJS:

import { Kafka, logLevel } from 'kafkajs';

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092'],
});

const producer = kafka.producer({
  idempotent: true,
  transactionTimeout: 30000,
  maxInFlightRequests: 5,
});

await producer.connect();

await producer.send({
  topic: 'orders',
  messages: [
    {
      key: 'order-123',
      value: JSON.stringify({
        orderId: '123',
        customerId: 'cust-456',
        amount: 99.99,
        timestamp: new Date().toISOString(),
      }),
    },
  ],
});

await producer.disconnect();

In .NET with Confluent.Kafka, the equivalent is:

using Confluent.Kafka;

var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    ClientId = "my-app",
    EnableIdempotence = true,
    Acks = Acks.All,
    MaxInFlight = 5,
};

using var producer = new ProducerBuilder<string, string>(config).Build();

var result = await producer.ProduceAsync("orders", new Message<string, string>
{
    Key = "order-123",
    Value = JsonSerializer.Serialize(new
    {
        orderId = "123",
        customerId = "cust-456",
        amount = 99.99,
        timestamp = DateTime.UtcNow,
    }),
});

producer.Flush();

Both examples are minimal, but they show the pattern: pass idempotent: true (KafkaJS) or EnableIdempotence = true (.NET), and let the client handle the rest.

Kafka deduplicates retries per partition

Once you enable idempotence, here's what happens under the hood:

1. The producer assigns a sequence number (0, 1, 2, ...) to each message for a given partition.
2. The broker stores the last sequence number it saw from each producer.
3. If a message arrives out of order or as a duplicate, the broker compares the sequence number and rejects it if it's already been seen.
4. The producer automatically retries without user intervention; the retry is silently deduplicated server-side.

This deduplication is per partition, not per topic. That's a critical distinction. If you send a message to partition 0 and then partition 1, they each maintain their own sequence counter. This design choice keeps the overhead low and scales well across many partitions.

Let me show a real scenario. Imagine you're building a SaaS billing system where monthly invoices are generated:

async function publishInvoiceEvent(customerId: string) {
  const invoice = {
    invoiceId: `inv-${Date.now()}`,
    customerId,
    amount: 150.00,
    dueDate: new Date(Date.now() + 30 * 24 * 60 * 60 * 1000).toISOString(),
    createdAt: new Date().toISOString(),
  };

  // This will be sent to a specific partition based on customerId
  await producer.send({
    topic: 'billing.invoices',
    messages: [
      {
        key: customerId, // Same key = same partition
        value: JSON.stringify(invoice),
      },
    ],
  });
}

// Network hiccup causes a retry, but Kafka deduplicates it.
await publishInvoiceEvent('cust-001');

Even if the network flakes and the producer retries five times, Kafka will only write the invoice once to the partition assigned to cust-001. The consumer sees exactly one invoice event, and your billing system processes it once.

Requires acks=all

Idempotence doesn't work in isolation. It requires acks=all (also called acks=-1). This ensures that the message is replicated to all in-sync replicas (ISRs) before the producer receives an acknowledgment. Without full replication, a broker crash could lose an already-acknowledged message, and the producer would retry and duplicate it elsewhere.

Here's why they're coupled: the broker needs to be certain that the message is durably stored before it can say "yes, I've received and deduplicated this." Partial replication introduces risk.

Update your configurations to include this:

// KafkaJS
const producer = kafka.producer({
  idempotent: true,
  transactionTimeout: 30000,
  maxInFlightRequests: 5,
  // Note: KafkaJS sets acks=all automatically when idempotent=true
});

// .NET Confluent.Kafka
var config = new ProducerConfig
{
    BootstrapServers = "localhost:9092",
    EnableIdempotence = true,
    Acks = Acks.All, // Explicitly required
    MaxInFlight = 5,
};

The latency trade-off is real. acks=all means waiting for replication, which adds ~10–50 ms depending on your network and replication factor. In a high-throughput trading system or ride-hailing platform, this is usually acceptable because the alternative—duplicates or data loss—is far more costly. But benchmark it in your environment before rolling out to production.

Default in modern Kafka clients

One of the best news is that idempotence is the default in most modern Kafka client libraries—or at least, it's strongly recommended and easy to enable by default. This represents a shift in the Kafka ecosystem toward safer defaults.

KafkaJS: As of version 2.0+, idempotence is enabled by default for idempotent producers. You don't have to do anything special; just use the library normally and you're safe.

librdkafka (C/C++): The enable.idempotence property defaults to false for backward compatibility, but is recommended to be set to true. Confluent's .NET, Python, and Go clients all inherit from librdkafka, so you should explicitly enable it.

Apache Kafka's Java client: In Kafka 3.0+, enable.idempotence=true is the default. Older versions default to false

The practical takeaway: verify your client version and your actual configuration. Don't assume; log it or check the Kafka metrics to confirm idempotence is active.

// Always log your producer config for debugging
const config = {
  idempotent: true,
  transactionTimeout: 30000,
  maxInFlightRequests: 5,
};

console.log('Producer config:', config);

const producer = kafka.producer(config);