Delivery semantics explained: what developers should know

Delivery semantics explained: choosing the right guarantee for your system

When you build a system that moves data from Kafka into a database, sends events to WebSocket clients, or chains microservices together, one critical question emerges: what happens if a message gets processed, but the acknowledgment never makes it back? Over five years building production platforms, I've learned that delivery semantics explained is less about theory and more about understanding the real cost of each choice. In this post, we'll dig into the three guarantees Kafka offers, when each makes sense, and how to design systems that handle failure gracefully.

The three guarantees: semantics versus the means to achieve them

Kafka gives you three delivery semantics. It's important to separate what each means (the semantic guarantee) from how you achieve it (the configuration and code).

I've seen teams confuse the semantic with the mechanism. At-most-once doesn't mean "fast"—it means "willing to lose data." At-least-once doesn't mean "slow"—it means "willing to handle duplicates." Exactly-once in Kafka is powerful, but it has hard boundaries that most developers overlook.

At-most-once: when losing data is acceptable

At-most-once means you commit your offset before processing the message. If your consumer crashes mid-processing, that message is gone forever.

// At-most-once: commit first, process later
kafkaConsumer.on('message', async (message) => {
  // Commit offset immediately
  await kafkaConsumer.commitOffsets([
    { topic: 'events', partition: 0, offset: message.offset + 1 }
  ]);
  
  // Now process
  try {
    await processEvent(message.value);
  } catch (err) {
    // Too late—offset already committed. Message lost.
    console.error('Processing failed but offset committed:', err);
  }
});

This guarantees no duplicates because once an offset is committed, it's treated as processed forever. But if something fails between the commit and the actual processing, the message vanishes.

Use at-most-once for analytics and metrics. If you lose one click or one pageview, your dashboard is still useful. I've shipped monitoring systems where duplicate metrics would actually skew results worse than missing them. Another real case: streaming logs to S3 for archival. One missing log entry in millions won't break compliance.

But never use it for financial transactions, inventory changes, or anything that auditors will ask about later.

At-least-once: the default production choice

At-least-once is the opposite: process the message first, then commit the offset. If you crash after processing but before the commit, the message will be reprocessed—but you'll never lose it.

// At-least-once: process first, commit after
kafkaConsumer.on('message', async (message) => {
  try {
    // Process the message first
    await processEvent(message.value);
    
    // Only commit after successful processing
    await kafkaConsumer.commitOffsets([
      { topic: 'events', partition: 0, offset: message.offset + 1 }
    ]);
  } catch (err) {
    // Offset not committed. Message will be retried.
    console.error('Processing failed, will retry:', err);
    throw err;
  }
});

The cost is duplicates. Under network partitions, broker failures, or consumer restarts, you might process the same message twice. That's where idempotent consumers come in.

Pairing at-least-once with idempotent design

An idempotent consumer produces the same result whether a message is processed once or ten times. The simplest pattern: store the event ID and check it before processing.

const db = new PrismaClient();

kafkaConsumer.on('message', async (message) => {
  try {
    const event = JSON.parse(message.value);
    const eventId = event.id; // Assume events have unique IDs
    
    // Check if we've already processed this event
    const existing = await db.processedEvents.findUnique({
      where: { eventId }
    });
    
    if (existing) {
      // Already processed. Idempotent: do nothing and commit.
      await kafkaConsumer.commitOffsets([
        { topic: 'events', partition: 0, offset: message.offset + 1 }
      ]);
      return;
    }
    
    // First time seeing this event. Process it.
    await processEvent(event);
    
    // Record that we've processed it
    await db.processedEvents.create({
      data: { eventId, processedAt: new Date() }
    });
    
    await kafkaConsumer.commitOffsets([
      { topic: 'events', partition: 0, offset: message.offset + 1 }
    ]);
  } catch (err) {
    console.error('Consumer error:', err);
    throw err;
  }
});

Now, if the consumer crashes and reprocesses the message, the findUnique check catches it. You get at-least-once delivery with exactly-once effect.

In a ride-hailing system I worked on, drivers submitted trip completion events with a unique `tripId`. We stored every processed `tripId` in Redis with a short TTL. Duplicates were silently dropped at the consumer level. The driver still got paid exactly once, even if Kafka replayed the event.

This is the pattern I recommend for 90% of production services. It's simple, reliable, and doesn't require the overhead of Kafka transactions.

Exactly-once: the powerful but narrow guarantee

Kafka's exactly-once semantic is genuinely useful, but it has a hard boundary: it covers Kafka-to-Kafka operations only. It does not cover side effects like sending emails, calling external APIs, or charging credit cards.

Exactly-once requires three things:

1. Idempotent producer (enable.idempotence=true): ensures Kafka never stores duplicate messages from the producer
2. Transactional writes: the consumer processes messages and writes results to Kafka in an atomic transaction
3. Isolation level (isolation.level=read_committed): consumers only see committed data

// Producer: idempotent configuration
const producer = kafka.producer({
  idempotentWrites: true, // Enable idempotent producer
  transactionalId: 'my-producer-1', // For transactional writes
});

// Consumer: read_committed isolation
const consumer = kafka.consumer({
  groupId: 'my-group',
  sessionTimeout: 30000,
});

// Set isolation level
await consumer.subscribe({ 
  topic: 'input-topic',
  fromBeginning: true
});

// Process and produce within a transaction
const admin = kafka.admin();
await admin.createTopics({
  topics: [{ name: 'output-topic' }],
});

consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    // Produce output message—both offset commit and produce happen
    // in a single transaction
    await producer.sendBatch({
      topicMessages: [
        {
          topic: 'output-topic',
          messages: [{ value: transformMessage(message.value) }],
        },
      ],
    });
  },
});

In this setup, the consumer reads a message, transforms it, and publishes the result to another topic. Both the offset update (consumer) and the produce (to output-topic) happen atomically. No gaps, no duplicates.

But here's the catch: exactly-once does not extend outside Kafka. If your consumer also sends an email, the guarantee breaks. The message might be processed exactly once in Kafka, but the email could be sent twice if the network hiccups.

I've seen teams spend weeks setting up exactly-once Kafka configurations only to realize their downstream HTTP calls to a payment API need the same idempotency logic anyway. Don't fall into that trap.

Exactly-once has boundaries: side effects still need idempotency

Let's say you're processing order events and need to send a confirmation email to the customer. Even with exactly-once semantics in Kafka, the email can be sent twice:

// Exactly-once in Kafka, but side effect is at-least-once
consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const order = JSON.parse(message.value);
    
    // This is atomic in Kafka
    await producer.send({
      topic: 'order-confirmed',
      messages: [{ value: JSON.stringify(order) }],
    });
    
    // But this is NOT covered by exactly-once
    try {
      await sendEmailViaSES(order.customerEmail, 'Order confirmed');
    } catch (err) {
      // If this fails after Kafka transaction commits,
      // the message is considered processed.
      // Consumer won't retry. Email might get sent twice
      // when we manually rewind and replay.
      console.error('Email failed:', err);
    }
  },
});

To handle side effects safely, treat them as at-least-once operations. Make your email service idempotent by storing a record of sent emails and checking it before sending again.

Design for duplicates: store event IDs or use upserts

Duplicates are a feature of at-least-once systems, not a bug. Design for them from the start.

Pattern 1: Event ID deduplication

Store processed event IDs in a fast store (Redis, DynamoDB, or even a database table with an index).

const redis = new Redis();

consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const event = JSON.parse(message.value);
    const dedupeKey = `processed:${event.id}`;
    
    // Check if we've processed this before (O(1) with Redis)
    const alreadyProcessed = await redis.exists(dedupeKey);
    
    if (alreadyProcessed) {
      await consumer.commitOffsets([
        { topic, partition, offset: message.offset + 1 },
      ]);
      return;
    }
    
    // Process the event
    await handleEvent(event);
    
    // Mark as processed with a TTL (keep for 24 hours for safety)
    await redis.setex(dedupeKey, 86400, '1');
    
    await consumer.commitOffsets([
      { topic, partition, offset: message.offset + 1 },
    ]);
  },
});

Pattern 2: Upserts

Use database upsert operations (INSERT ... ON CONFLICT DO UPDATE) to make writes idempotent at the database level.

// Every write is idempotent because of the unique constraint
await db.inventory.upsert({
  where: { sku: product.sku },
  update: {
    quantity: { decrement: product.quantity },
    lastEventId: product.eventId,
  },
  create: {
    sku: product.sku,
    quantity: product.initialQuantity - product.quantity,
    lastEventId: product.eventId,
  },
});

If the same product update event is processed twice, the second upsert just re-applies the same change. No duplicate deduction, no lost updates.

This is especially powerful in e-commerce. I've used upserts to handle inventory updates from multiple Kafka topics. Duplicates become harmless because the final state is always the same.

Common mistakes

• Confusing "exactly-once Kafka" with "exactly-once system". Kafka transactions are Kafka-only. Your HTTP calls, database writes to external systems, and webhooks still need idempotency logic. I've seen teams enable exactly-once configuration and then wonder why they're still getting duplicate emails.
• Choosing at-most-once for data that matters. It's tempting because it's simpler, but losing an order, payment, or inventory update is a data integrity nightmare. Stay with at-least-once + idempotency.
• Not storing event IDs. If your messages don't include a stable, unique ID, you can't deduplicate properly. Make sure every event has an `id` field set by the producer.
• Ignoring network partitions. Even with at-least-once, if your consumer can't reach the broker to commit, it will reprocess. This is correct behavior, but you need idempotent handlers in place.
• Using exactly-once to avoid idempotency altogether. It's still worth implementing idempotency even with exactly-once because operational replays, manual debugging, and code changes can still cause duplicates in practice.

Key takeaways

• At-most-once means "willing to lose data"—use only for non-critical analytics and logs.
• At-least-once is the standard choice: process first, commit after, then deduplicate with event IDs or upserts.
• Pair at-least-once delivery with idempotent consumers to get exactly-once effect without the complexity.
• Kafka's exactly-once semantic covers only Kafka-to-Kafka operations. Side effects still need idempotency.
• Design your systems expecting duplicates. Store event IDs, use Redis for quick checks, or rely on database upserts.
• The real cost isn't choosing the semantic—it's implementing idempotency correctly. That's what matters in production.