Message IDs and deduplication: best practices explained

Message IDs and deduplication

When you're building a real-time messaging system that spans client and server over WebSockets and Kafka, the worst thing that can happen is silent duplication: a user hits send, the message arrives on the server, but the network drops before the client gets the confirmation. The user retries—now you have two messages. Message IDs and deduplication is the pattern that stops this dead. By assigning each message a unique ID on the client side and teaching the server to reject duplicates, you turn unreliable networks into a reliable, exactly-once delivery experience.

Why deduplication matters in distributed systems

In my experience building e-commerce and messaging platforms, the database doesn't care about your intentions—it cares about what you send it. If a message travels from browser to server, gets persisted to Kafka, but the server crashes before sending an acknowledgment, the client doesn't know if it succeeded or failed. The sensible thing is to retry. But without deduplication, that retry becomes a duplicate message in your system.

This is especially critical in scenarios like:

• Chat applications where duplicate messages are visible and confusing to users
• Payment systems where duplicate transactions have serious financial consequences
• Ride-hailing platforms where duplicate booking requests can oversell a driver
• Trading systems where duplicate orders can blow through position limits

Deduplication guarantees that a message with the same ID, sent multiple times, is processed exactly once on the server side.

Client generates a unique message ID

Every message that leaves the client must carry a unique identifier. This ID is generated client-side before sending, and it stays with the message through its entire lifecycle: from the browser, through WebSocket, into Kafka, and into your database.

In practice, a UUID v4 is reliable and collision-free. Here's how I'd generate and attach it in a React client using WebSocket:

import { v4 as uuidv4 } from 'uuid';

interface OutgoingMessage {
  messageId: string;
  text: string;
  conversationId: string;
  timestamp: number;
}

function sendMessage(text: string, conversationId: string) {
  const messageId = uuidv4();
  const timestamp = Date.now();

  const message: OutgoingMessage = {
    messageId,
    text,
    conversationId,
    timestamp,
  };

  // Store locally with "sending" state before transmitting
  storeMessageLocally(messageId, { ...message, status: 'sending' });

  // Send over WebSocket
  socket.emit('message:send', message);

  // Set a timeout to retry if no ack arrives
  const timeoutId = setTimeout(() => {
    console.warn(`No ack for ${messageId}, retrying...`);
    socket.emit('message:send', message);
  }, 5000);

  // Listen for acknowledgment
  socket.once(`message:ack:${messageId}`, () => {
    clearTimeout(timeoutId);
    updateMessageStatus(messageId, 'sent');
  });
}

Key points here: the messageId is generated once and reused if the client needs to retry. It's never regenerated between retries—that would defeat the purpose. The client also stores the message locally with a "sending" status, which gives immediate feedback to the user while the network request is in flight.

Server ignores duplicate IDs

On the server side, every incoming message must be checked against a deduplication store before being processed. This store is typically a Redis cache or a database index, and it tracks which message IDs have already been handled.

Here's a NestJS example using Redis and PostgreSQL:

import { Injectable } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { Repository } from 'typeorm';
import { Redis } from 'ioredis';
import { InjectRedis } from '@nestjs-modules/ioredis';

@Injectable()
export class MessageService {
  constructor(
    @InjectRepository(Message)
    private messageRepository: Repository<Message>,
    @InjectRedis() private redis: Redis,
  ) {}

  async handleIncomingMessage(
    messageId: string,
    text: string,
    conversationId: string,
    userId: string,
  ): Promise<{ status: 'created' | 'duplicate'; messageId: string }> {
    // Check Redis for recent duplicate (short TTL, e.g., 1 hour)
    const dedupeKey = `message:dedup:${messageId}`;
    const existing = await this.redis.get(dedupeKey);

    if (existing) {
      console.log(`Duplicate detected: ${messageId}`);
      return { status: 'duplicate', messageId };
    }

    // Create the message in the database
    const message = this.messageRepository.create({
      id: messageId, // Use the client-generated ID as primary key
      text,
      conversationId,
      userId,
      createdAt: new Date(),
    });

    await this.messageRepository.save(message);

    // Mark as processed in Redis (TTL = 1 hour)
    await this.redis.setex(dedupeKey, 3600, '1');

    return { status: 'created', messageId };
  }
}

Notice the two-layer defense: Redis for fast, in-memory deduplication checks, and the database for persistent dedup verification. If a message arrives twice within the same hour, Redis catches it instantly. If Redis is cleared or restarted, the database primary key constraint (using messageId as the primary key) prevents accidental duplicates from being created.

The response always includes the messageId so the client knows exactly which message was acknowledged.

Show sent, delivered and read states

Deduplication works best when paired with explicit message state tracking. Users want to know: did my message actually go through? Has it been read? This is where the "sent," "delivered," and "read" pattern comes in.

• Sent: The server received and persisted the message
• Delivered: The recipient's client fetched and displayed the message
• Read: The recipient has viewed the message (optional but valuable)

Here's how I'd store and track these states:

interface MessageState {
  messageId: string;
  status: 'sending' | 'sent' | 'delivered' | 'read';
  createdAt: number;
  sentAt?: number;
  deliveredAt?: number;
  readAt?: number;
}

// Server-side, after persisting the message:
async function publishMessageEvent(messageId: string, conversationId: string) {
  const event = {
    type: 'message.created',
    messageId,
    conversationId,
    status: 'sent',
    timestamp: Date.now(),
  };

  // Emit event to all subscribers via WebSocket
  io.to(`conversation:${conversationId}`).emit('message:status', event);

  // Also publish to Kafka for persistent log
  await kafkaProducer.send({
    topic: 'conversation-events',
    messages: [{ key: conversationId, value: JSON.stringify(event) }],
  });
}

On the client, when a message is received and rendered, it sends a "delivered" event back to the server. When the user reads it (e.g., scrolls into view), it sends a "read" event. Each state change is its own event, and deduplication logic applies to the original message, not the state-change events.

Retry safely on reconnect

The real test of deduplication is what happens when a WebSocket disconnects and reconnects. The client needs to retry all "sending" messages without creating duplicates.

Here's a reconnection handler that does this safely:

socket.on('connect', async () => {
  console.log('WebSocket connected');

  // Fetch all locally-stored messages in "sending" state
  const pendingMessages = await getLocalMessages('sending');

  for (const msg of pendingMessages) {
    console.log(`Retrying ${msg.messageId}`);

    // Send again with the same messageId
    socket.emit('message:send', {
      messageId: msg.messageId,
      text: msg.text,
      conversationId: msg.conversationId,
      timestamp: msg.timestamp,
    });

    // Set up the ack listener again
    socket.once(`message:ack:${msg.messageId}`, () => {
      updateMessageStatus(msg.messageId, 'sent');
    });
  }

  // For messages we believe are already sent, sync state with server
  const sentMessages = await getLocalMessages('sent');
  if (sentMessages.length > 0) {
    socket.emit('message:sync', {
      messageIds: sentMessages.map(m => m.messageId),
    });
  }
});

The key insight: we retry with the exact same messageId that was generated on the first attempt. The server's dedup logic handles the duplicate without flinching. This is safe, predictable, and testable.

Common mistakes

• Regenerating the messageId on retry: If you call uuidv4() again when the first send fails, you've created two distinct messages in the system. Always store and reuse the original ID.
• Relying only on Redis for deduplication: Redis data can be lost. If you're building anything that matters (payments, orders, messages), persist dedup checks to a database too.
• Using the same messageId across different message types: A messageId should be unique per outgoing message. Don't reuse it for typing indicators or other ephemeral events.
• Forgetting to communicate messageId to Kafka consumers: If you publish to Kafka without the messageId, downstream consumers can't deduplicate. Always include it in the event payload.
• Setting dedup TTLs too short: If your Redis TTL is 5 minutes but your user retries after 10 minutes, you've lost the dedup guarantee. Use a TTL that aligns with your retry window and data retention policy (typically 1–24 hours).

Key takeaways

• Every outgoing message gets a unique, client-generated ID (UUID v4) before it's sent
• The server checks a fast dedup store (Redis) before processing; if the ID exists, it returns "duplicate" without re-processing
• Pair deduplication with explicit message states (sending, sent, delivered, read) so the client and user always know where a message stands
• On reconnect, retry all pending messages with their original IDs; the server's dedup layer silently absorbs the duplicates
• Back up your dedup store with a database constraint (use messageId as primary key) so even cache loss doesn't break the guarantee
• Dedup TTLs should match your retry window and data retention needs, typically 1 hour to 24 hours