Acknowledgement-based delivery: best practices explained

Acknowledgement-based delivery: best practices explained. When a message must arrive.

Building reliability into real-time systems

In distributed systems handling real-time messaging, the difference between "probably delivered" and "definitely delivered" can cost you customers. Acknowledgement-based delivery is the pattern that bridges this gap, ensuring critical messages actually reach their destination even when networks fail or clients disconnect. We'll walk through the mechanics, patterns, and pitfalls that teams encounter when implementing this in production Kafka and WebSocket environments.

Why acknowledgement-based delivery matters

Consider a ride-hailing platform where a driver accepts a ride request. The backend publishes an event to Kafka, forwards it over WebSocket to the mobile app, and waits. If the message never reaches the client, the driver never sees the request. Without acknowledgement, you have no way of knowing whether the driver's app received it or whether it was lost in transit.

Acknowledgement-based delivery creates a contract: the server sends a message with a unique identifier, the client confirms receipt by sending back that identifier, and the server only considers the exchange complete once that acknowledgement arrives. This transforms a fire-and-forget system into one where you can guarantee message delivery with confidence.

Server sends with a message ID

Every message flowing over your WebSocket connection needs a unique identifier that both client and server can reference. This ID becomes the cornerstone of your acknowledgement system—it's how the server tracks which messages are pending and which have been received.

On the server side, generate a message ID when you pull from Kafka or create a new event. Common approaches include UUIDs, incrementing sequences per connection, or timestamp-based IDs. The key is that the ID must be unique within the scope of that client connection and stable enough that you can reliably match it against incoming acknowledgements.

interface OutgoingMessage {
  id: string;
  type: string;
  payload: unknown;
  timestamp: number;
}

interface PendingMessage {
  id: string;
  message: OutgoingMessage;
  sentAt: number;
  retryCount: number;
}

class WebSocketHandler {
  private pendingMessages: Map = new Map();

  async sendMessage(client: WebSocket, message: unknown, type: string): Promise {
    const messageId = crypto.randomUUID();
    const outgoing: OutgoingMessage = {
      id: messageId,
      type,
      payload: message,
      timestamp: Date.now(),
    };

    this.pendingMessages.set(messageId, {
      id: messageId,
      message: outgoing,
      sentAt: Date.now(),
      retryCount: 0,
    });

    client.send(JSON.stringify(outgoing));
    return messageId;
  }
}

Keep the message ID visible in your wire protocol so the client can easily extract it from the JSON payload. Avoid burying it deep in nested structures—the client needs to acknowledge quickly, and every millisecond of processing adds latency.

Client acknowledges receipt

The client's job is straightforward: receive the message, process it if needed, and send back an acknowledgement containing the message ID. This acknowledgement is a separate message flowing in the opposite direction, confirming that the client successfully parsed and stored the data.

In practice, the client should acknowledge after it has reliably stored the message locally, not before. On a mobile app with a local database, this means writing to SQLite or Realm before sending the ACK. On a browser, this might mean storing in IndexedDB if the message is critical. The acknowledgement is your guarantee that the message will survive a process restart.

interface AcknowledgementMessage {
  type: 'ack';
  messageId: string;
  receivedAt: number;
}

class WebSocketClient {
  private socket: WebSocket;
  private localDb: Database;

  constructor(url: string) {
    this.socket = new WebSocket(url);
    this.socket.onmessage = (event) => this.handleMessage(event.data);
  }

  private async handleMessage(data: string): Promise {
    const incoming = JSON.parse(data);

    if (incoming.type === 'ack') {
      // Server is acknowledging our ACK; we can clean up
      return;
    }

    // Store the message in local DB first
    await this.localDb.messages.insert({
      id: incoming.id,
      type: incoming.type,
      payload: incoming.payload,
      receivedAt: Date.now(),
    });

    // Now send acknowledgement back
    const ack: AcknowledgementMessage = {
      type: 'ack',
      messageId: incoming.id,
      receivedAt: Date.now(),
    };

    this.socket.send(JSON.stringify(ack));
  }
}

This pattern is critical on mobile platforms where connectivity is unreliable. A user might receive a message, acknowledge it to your server, then lose signal before the app processes the event. By persisting before acknowledging, you ensure the message survives even if the connection drops immediately after the ACK is sent.

Server retries unacknowledged messages

The server maintains a list of pending messages—those sent but not yet acknowledged. On a timer or when certain conditions are met, the server retries any message that hasn't been acknowledged within a reasonable window. This is where exponential backoff becomes your friend.

A typical retry strategy waits 1 second for the first acknowledgement, then 2 seconds, 4 seconds, 8 seconds, capping at some maximum like 60 seconds. After a fixed number of retries (often 3–5), you might give up and log an alert for manual investigation. The exact numbers depend on your use case—a financial transaction might retry aggressively for several minutes, while a notification might give up after 30 seconds.

const RETRY_CONFIG = {
  maxRetries: 5,
  initialDelayMs: 1000,
  maxDelayMs: 60000,
  backoffFactor: 2,
};

class MessageRetryManager {
  private pendingMessages: Map = new Map();
  private retryTimers: Map = new Map();

  scheduleRetry(messageId: string): void {
    const pending = this.pendingMessages.get(messageId);
    if (!pending) return;

    const nextRetry = Math.min(
      RETRY_CONFIG.initialDelayMs *
        Math.pow(RETRY_CONFIG.backoffFactor, pending.retryCount),
      RETRY_CONFIG.maxDelayMs
    );

    if (pending.retryCount >= RETRY_CONFIG.maxRetries) {
      console.error(`Message ${messageId} exceeded max retries`);
      this.pendingMessages.delete(messageId);
      return;
    }

    const timer = setTimeout(() => {
      pending.retryCount++;
      pending.sentAt = Date.now();
      // Resend the message to the client
      this.resendMessage(pending);
      this.scheduleRetry(messageId);
    }, nextRetry);

    this.retryTimers.set(messageId, timer);
  }

  acknowledgeMessage(messageId: string): void {
    const timer = this.retryTimers.get(messageId);
    if (timer) {
      clearTimeout(timer);
      this.retryTimers.delete(messageId);
    }
    this.pendingMessages.delete(messageId);
  }

  private resendMessage(pending: PendingMessage): void {
    // Logic to resend to the connected client
    console.log(`Retrying message ${pending.id} (attempt ${pending.retryCount})`);
  }
}

The server must track which connection each pending message belongs to. If a client disconnects and reconnects, you might reassign pending messages to the new connection. Conversely, if a message has been retried 5 times and still no ACK, it's worth logging for inspection—it might indicate a bug in the client's acknowledgement logic or a persistently bad network condition.

Store pending messages for offline users

The real challenge emerges when a user is offline. Their WebSocket is closed, so you can't retry over that channel. You need a separate store—usually a database—that persists pending messages until the client comes back online and drains the backlog.

This is where Kafka becomes invaluable. Rather than storing pending messages in your application's memory or a separate queue, you can leverage Kafka topics with compacted retention. Each user has a queue of unacknowledged messages; when they reconnect, the server queries that queue and replays all pending messages in order.

In a typical flow: the server publishes a message to Kafka, publishes it over WebSocket if the client is online, and stores a reference to the message in a "pending" table. When an ACK arrives, the server marks that message as acknowledged in the database. When a user reconnects, the server queries all non-acknowledged messages for that user and resends them.

interface UserMessageQueue {
  userId: string;
  messageId: string;
  messageType: string;
  payload: string;
  createdAt: Date;
  acknowledgedAt?: Date;
}

class OfflineMessageStore {
  private db: Database;
  private kafka: KafkaProducer;

  async storeMessageForUser(
    userId: string,
    message: OutgoingMessage
  ): Promise {
    // Store in database for offline persistence
    await this.db.userMessageQueues.insert({
      userId,
      messageId: message.id,
      messageType: message.type,
      payload: JSON.stringify(message.payload),
      createdAt: new Date(),
    });

    // Also publish to Kafka for event sourcing
    await this.kafka.send({
      topic: `user-messages-${userId}`,
      messages: [
        {
          key: userId,
          value: JSON.stringify(message),
          headers: { messageId: message.id },
        },
      ],
    });
  }

  async getPendingMessages(userId: string): Promise {
    const rows = await this.db.userMessageQueues.find({
      userId,
      acknowledgedAt: null,
    });

    return rows.map((row) => ({
      id: row.messageId,
      type: row.messageType,
      payload: JSON.parse(row.payload),
      timestamp: row.createdAt.getTime(),
    }));
  }

  async markAsAcknowledged(userId: string, messageId: string): Promise {
    await this.db.userMessageQueues.update(
      { userId, messageId },
      { acknowledgedAt: new Date() }
    );
  }
}

When reconnection happens, send all pending messages in a single burst. The client's local store combined with the server's database creates two layers of durability. Even if both the client and server lose data, Kafka retains a complete audit trail.

Common mistakes

• Acknowledging before persisting: Servers often mark a message as acknowledged the instant it arrives, before checking if the message ID actually makes sense or if the client can handle it. Always acknowledge after you've validated and processed.
• Forgetting to deduplicate on reconnect: When a client reconnects and you replay pending messages, the client might already have some of them from before the disconnect. Implement deduplication using the message ID and a local store so the client doesn't process the same event twice.
• No upper bound on retry delays: Without a maximum retry delay, exponential backoff can lead to messages being retried only once per hour. Cap it at a sensible value like 60 seconds.
• Storing messages in memory only: If your server crashes, all pending messages in memory vanish. Always persist to a database or Kafka topic.
• Ignoring ACK ordering: If messages must be processed in order, an out-of-order ACK can lead to gaps. Ensure your client acknowledges messages sequentially or that your server enforces ordering when replaying.
• Timeout values that are too aggressive: Mobile networks can be slow. Setting a 500ms acknowledgement timeout will cause unnecessary retries. Aim for 5–10 seconds as a starting point and adjust based on monitoring.

Key takeaways

• Acknowledgement-based delivery provides a contract: every message carries a unique ID, the client confirms receipt, and the server retries until acknowledged.
• Always persist messages client-side before sending an ACK, ensuring they survive disconnections or crashes immediately after acknowledgement.
• Use exponential backoff with a reasonable cap (e.g., 1s, 2s, 4s, 8s, capped at 60s) to balance quick retries with avoiding server overload.
• Store unacknowledged messages in a database or Kafka topic so offline users get a complete backlog when they reconnect.
• Implement deduplication on the client side to prevent processing the same message twice after reconnection.
• Monitor acknowledgement latency and retry rates—high retry counts often signal a deeper issue like a client-side bug or systematic connectivity problems.
• Test disconnection scenarios thoroughly: sudden drops, gradual timeouts, and reconnections with a large backlog all behave differently in production.