Outbox and change data capture: best practices explained

Outbox and change data capture: best practices explained

Your service just wrote an order to the database, but the Kafka message never arrived. Minutes later, the warehouse system times out waiting for shipment details. This scenario repeats across distributed teams until someone builds a workaround that feels like a patch over a deeper wound. The real problem isn't bad luck—it's trying to keep two systems in sync when they're not connected by a transaction boundary. In this article, we'll walk through how the outbox pattern combined with change data capture (CDC) gives you reliable events straight from your database without the coordination headaches.

The dual-write problem

Most teams start with a straightforward approach: write data to PostgreSQL, then publish a message to Kafka. The code feels simple and the happy path works every time.

async function createOrder(order: Order) {
  // Write to database
  await db.orders.insert(order);
  
  // Publish to Kafka
  await kafka.producer.send({
    topic: 'orders',
    messages: [{ value: JSON.stringify(order) }]
  });
}

Then reality kicks in. The database write succeeds, but the Kafka producer times out. Or Kafka accepts the message but the database transaction rolls back due to a constraint violation. Or the application crashes between the two calls. Now you have an order in the database that the warehouse never heard about, or a Kafka message for an order that doesn't exist. Both scenarios create silent data corruption that cascades through your system.

The core issue is that database transactions and message brokers don't share a transaction boundary. You cannot atomically write to both. You can add retries, circuit breakers, and dead-letter queues, but none of these address the fundamental problem: if either operation fails in isolation, you're left with inconsistent state.

This is where the outbox pattern enters. Instead of publishing directly to Kafka, you write the event to a table in the same database transaction as your business data. The database guarantees atomicity; either both the order and the event row are committed together, or neither is. Then a separate process—powered by CDC—tails your database log and publishes those events to Kafka.

Outbox table: write events alongside your data

An outbox table lives in your application database and acts as a staging area for events. When you perform a state change (like creating an order), you write a row to the outbox in the same transaction. The schema is simple:

The published_at column tracks whether the CDC process has published this event to Kafka. Initially it's null; once Debezium reads the change and publishes it, the column is updated.

Here's what the application code looks like:

async function createOrder(order: Order) {
  await db.transaction(async (trx) => {
    // Write the order
    await trx.table('orders').insert({
      id: order.id,
      customer_id: order.customerId,
      total: order.total,
      status: 'pending'
    });

    // Write the outbox event in the same transaction
    await trx.table('outbox').insert({
      id: crypto.randomUUID(),
      aggregate_id: order.id,
      event_type: 'OrderCreated',
      payload: JSON.stringify(order),
      created_at: new Date()
    });
  });
}

Notice the transaction wrapper. Both INSERT statements succeed or both fail together. If anything goes wrong—network error, constraint violation, deployment crash—the entire transaction rolls back, and the outbox row is never created. There's no window of inconsistency.

This approach scales to complex workflows. An order might move through multiple states: pending, confirmed, shipped, delivered. Each transition writes a new outbox row in the same transaction as the status update. The event log becomes an append-only record of everything that happened, which is invaluable for auditing, replayability, and debugging.

Debezium reads the log: CDC publishes outbox rows to Kafka

Once your outbox rows are safely in the database, a CDC connector needs to read them and publish them to Kafka. This is where Debezium comes in. Debezium is a distributed platform for change data capture that tails your database's transaction log (WAL on PostgreSQL, binlog on MySQL) and pushes changes downstream in real time.

Setting up Debezium for PostgreSQL is straightforward using Kafka Connect:

// POST /connectors HTTP/1.1
{
  "name": "outbox-connector",
  "config": {
    "connector.class": "io.debezium.connector.postgresql.PostgresConnector",
    "database.hostname": "postgres.prod",
    "database.port": "5432",
    "database.user": "replication_user",
    "database.password": "secret",
    "database.dbname": "orders_db",
    "database.server.name": "prod-orders",
    "schema.include.list": "public",
    "table.include.list": "public.outbox",
    "logical.decoding.plugin.name": "pgoutput",
    "publication.name": "outbox_publication",
    "plugin.name": "pgoutput"
  }
}

Debezium connects to PostgreSQL using logical replication, which requires minimal configuration and doesn't block production queries. It reads new outbox rows from the WAL and publishes them to a Kafka topic (typically prod-orders.public.outbox). Each message contains the row data and metadata about whether it was inserted, updated, or deleted.

Your event consumers subscribe to this topic and process the outbox rows. A typical event processor looks like this:

async function processOutboxEvent(kafkaMessage: any) {
  const change = kafkaMessage.payload.after; // The new row state
  
  if (!change) return; // Deleted row
  
  const event = {
    id: change.id,
    aggregateId: change.aggregate_id,
    type: change.event_type,
    data: JSON.parse(change.payload),
    timestamp: change.created_at
  };
  
  // Route to appropriate handler
  await handleOrderEvent(event);
  
  // Publish downstream (optional: to other services)
  await publisher.emit(event.type, event.data);
}

Because Kafka topics are immutable and indexed by log offset, consumers can resume from their last committed offset after a restart. They never miss events and never process the same event twice (assuming you implement idempotent handlers).

No polling needed: low latency without database load

Before CDC, teams often built polling solutions: a cron job or background worker that periodically queries the outbox table for unpublished rows, publishes them to Kafka, and marks them as published. This works but has serious drawbacks.

Polling is inherently latent. If your polling interval is 5 seconds, an event sits in the database for up to 5 seconds before publication. During peak traffic or with many services, the accumulated latency becomes unacceptable. Users see stale data; notifications arrive late; reconciliation jobs run on outdated state.

Polling also creates load on your database. Every 5 seconds (or however frequently you poll), the outbox query hits the same indexes, evicts hot data from the buffer pool, and competes with production traffic. Scale to hundreds of services all polling their own outbox tables, and your database becomes a bottleneck.

CDC eliminates both problems. Debezium doesn't poll; it reads directly from the transaction log. New events are published to Kafka within milliseconds of being committed to the database. The database sees no additional SELECT queries. The log read is sequential and nearly free compared to random index lookups.

In practice, teams report end-to-end latencies of 50–500ms from outbox write to Kafka publication, with zero polling overhead. This makes real-time features—inventory sync, fraud detection, push notifications—feasible on standard infrastructure.

Common mistakes

• Forgetting the published_at column. Without it, you can't distinguish which rows have been published. Debezium will re-publish the same events repeatedly, causing duplicate processing downstream. Always track publication state.
• Writing to outbox outside a transaction. If you insert the order in one transaction and the outbox row in another, you're back to the dual-write problem. Always wrap both in a single BEGIN...COMMIT block.
• Debezium consumer crashes without offset management. If your Kafka consumer crashes and you don't commit offsets, Debezium will re-publish old events. Use Kafka Consumer Groups with proper offset commits, and make your event handlers idempotent.
• Not cleaning up the outbox table. The outbox grows indefinitely if you don't archive or delete old rows. Implement a retention policy: delete rows older than 7 days or move them to an archive table for compliance.
• Assuming CDC is ordered across all tables. Debezium publishes changes per table in order, but cross-table ordering is not guaranteed. If you need global event ordering, add a sequence number or timestamp to the outbox payload.

Key takeaways

• The outbox pattern solves the dual-write problem by writing both data and events in a single database transaction, guaranteeing consistency.
• Change data capture (Debezium) tails your database log and publishes outbox rows to Kafka, eliminating the need for polling and reducing latency from seconds to milliseconds.
• CDC removes database load: log reads are sequential and negligible compared to polling queries.
• Always include a published_at column to track which events have been published and avoid duplicates.
• Make event handlers idempotent and manage Kafka consumer group offsets carefully to handle crashes gracefully.
• Implement outbox table retention and cleanup to prevent unbounded growth over time.