Outbox pattern with MongoDB: what developers should know

Introduction

When your application writes data to MongoDB and needs to publish events to Kafka, you face a distributed systems problem that's harder than it looks. The outbox pattern with MongoDB: what developers should know is about ensuring that data changes and their corresponding events never fall out of sync. Without a reliable mechanism, you risk events being lost if your publisher crashes, or events being sent multiple times if your writer succeeds but the publisher fails. Let's explore how to implement this pattern correctly in production systems.

The core challenge: atomic writes and reliable publishing

Most of us have written code that updates a document and then publishes an event. The problem is that these two operations aren't atomic across your database and your message broker. If the write succeeds but the broker is unreachable, the event is lost. If the broker accepts the event but your app crashes before acknowledging the write, you might publish duplicates. This gap between persistence and publishing is where data consistency breaks down.

The outbox pattern solves this by making both operations part of a single atomic transaction at the database level. Instead of writing to your main collection and then trying to send an event elsewhere, you write your business data and a corresponding outbox entry in one go. A separate publisher process then reads from the outbox and delivers events to Kafka, marking entries as sent only after confirmation.

Approach 1: Separate collections with transactions

The most common approach uses two collections: your main domain collection (orders, users, payments) and an outbox collection. You write to both in a MongoDB multi-document transaction.

Here's a practical example. Say you're running an e-commerce platform and need to create an order, then reliably notify downstream services:

async function createOrder(orderData) {
  const session = client.startSession();
  
  try {
    await session.withTransaction(async () => {
      // Write the order document
      const orderResult = await ordersCollection.insertOne(
        {
          _id: new ObjectId(),
          customerId: orderData.customerId,
          items: orderData.items,
          total: orderData.total,
          status: 'pending',
          createdAt: new Date()
        },
        { session }
      );

      // Write outbox entry in the same transaction
      await outboxCollection.insertOne(
        {
          _id: new ObjectId(),
          aggregateId: orderResult.insertedId,
          eventType: 'OrderCreated',
          eventData: {
            orderId: orderResult.insertedId,
            customerId: orderData.customerId,
            total: orderData.total
          },
          createdAt: new Date(),
          published: false
        },
        { session }
      );
    });
  } finally {
    await session.endSession();
  }
}

The transaction ensures both writes succeed or both fail. No partial states. The outbox entry acts as a durable todo list for your event publisher.

Your outbox collection might look like this schema:

This approach is straightforward and works well for systems with moderate event volume. It's also easy to debug: you can query the outbox collection to see what events are pending, what's been published, and what failed.

Approach 2: Embed outbox entries in the same document

For some use cases, you might want to store outbox entries alongside your domain data in the same document. This keeps related data together and can reduce the number of collections you need to manage.

Instead of a separate outbox collection, add an events array to your domain document:

async function createOrderWithEmbeddedEvents(orderData) {
  const session = client.startSession();
  
  try {
    await session.withTransaction(async () => {
      const orderId = new ObjectId();
      
      await ordersCollection.insertOne(
        {
          _id: orderId,
          customerId: orderData.customerId,
          items: orderData.items,
          total: orderData.total,
          status: 'pending',
          createdAt: new Date(),
          events: [
            {
              _id: new ObjectId(),
              eventType: 'OrderCreated',
              eventData: {
                orderId: orderId,
                customerId: orderData.customerId,
                total: orderData.total
              },
              createdAt: new Date(),
              published: false
            }
          ]
        },
        { session }
      );
    });
  } finally {
    await session.endSession();
  }
}

The advantage here is that the domain object and its events are always written together, reducing the chance of synchronization issues. The downside is that as events accumulate, the document size grows. MongoDB has a 16MB document size limit, so for high-event systems, this approach can become problematic.

A middle ground is to archive published events periodically, keeping only unpublished events in the array to manage document size.

The publisher: reading and sending to Kafka

Now you have outbox entries persisted. A separate service (or process) polls the outbox collection looking for unpublished events and sends them to Kafka. This decouples your write path from your publish path.

Here's a straightforward publisher loop:

async function publishOutboxEvents(kafkaProducer) {
  const batchSize = 50;
  
  while (true) {
    try {
      // Find unpublished events
      const unpublished = await outboxCollection
        .find({ published: false })
        .sort({ createdAt: 1 })
        .limit(batchSize)
        .toArray();

      if (unpublished.length === 0) {
        // No events to publish, wait before polling again
        await sleep(1000);
        continue;
      }

      // Send each event to Kafka
      const messages = unpublished.map(entry => ({
        topic: getTopicForEventType(entry.eventType),
        key: entry.aggregateId.toString(),
        value: JSON.stringify({
          eventType: entry.eventType,
          eventData: entry.eventData,
          timestamp: entry.createdAt
        })
      }));

      await kafkaProducer.sendBatch({ topicMessages: messages });

      // Mark as published
      const ids = unpublished.map(e => e._id);
      await outboxCollection.updateMany(
        { _id: { $in: ids } },
        { 
          $set: { 
            published: true,
            publishedAt: new Date()
          }
        }
      );

    } catch (error) {
      console.error('Outbox publisher error:', error);
      await sleep(5000); // Back off on error
    }
  }
}

Notice that we mark events as published after Kafka confirms receipt. This is crucial: if Kafka accepts the message and we crash before updating the database, we'll send the event again on restart. That's usually acceptable (Kafka consumers should be idempotent). But if we mark published first and then Kafka fails, we've lost the event.

Handling failures and idempotency

Kafka brokers can sometimes accept a message into the batch but fail to persist it if there's a broker failure immediately after. Your consumer may also process the same event twice due to rebalancing. This is why your downstream consumers must be idempotent: they should recognize and deduplicate events using the event ID or a combination of keys.

Add a unique event ID to your outbox entries to support this:

await outboxCollection.insertOne(
  {
    _id: new ObjectId(),
    eventId: crypto.randomUUID(), // Globally unique event identifier
    aggregateId: orderResult.insertedId,
    eventType: 'OrderCreated',
    eventData: { /* ... */ },
    createdAt: new Date(),
    published: false
  },
  { session }
);

Your Kafka consumers should then track seen event IDs in a separate deduplication store (Redis, a database table, etc.) to ensure they only process each event once.

Scaling: indexes and retention

As outbox entries accumulate, queries slow down. Create indexes to keep things fast:

// Index for the publisher's query
await outboxCollection.createIndex(
  { published: 1, createdAt: 1 },
  { name: 'idx_unpublished_events' }
);

// Index for debugging: find events by aggregate
await outboxCollection.createIndex(
  { aggregateId: 1 },
  { name: 'idx_by_aggregate' }
);

Over time, your outbox collection will grow. Implement a retention policy: after events have been published for a certain period (e.g., 30 days), archive or delete them. This keeps the active outbox collection lean and queries fast.

Common mistakes

• Marking published before Kafka confirms: If you update the outbox before Kafka successfully persists the message, you'll lose events on failure. Always wait for the broker's acknowledgment.
• Forgetting about embedded document size limits: If you embed events in your domain documents, the 16MB document limit will bite you eventually. Monitor document growth and have an archival strategy.
• No idempotency in consumers: Downstream services should deduplicate by event ID. Duplicate events are not just annoying; they can corrupt data if your consumer isn't prepared.
• Missing indexes on the outbox collection: Without a published + createdAt index, your publisher's polling query will do full collection scans and degrade under load.
• Not handling publisher crashes: If your publisher restarts, it should pick up where it left off. Design it to be stateless and poll for unpublished events on startup.

Key takeaways

• The outbox pattern uses atomic transactions to write business data and event records together, ensuring consistency.
• Separate outbox collections work well for most systems; embedded events in documents are simpler but hit size limits at scale.
• Your publisher should send to Kafka, then mark events as published—never the other way around.
• Implement idempotent consumers downstream to handle duplicate events gracefully.
• Create the right indexes and retention policies so your outbox collection doesn't become a performance bottleneck.
• This pattern eliminates the "dual-write" problem and is essential for building reliable event-driven systems on MongoDB.