Outbox pattern for payment events: a step-by-step guide

The reliability challenge in payment systems

Payment events are the backbone of reconciliation, customer notifications, and downstream billing processes. When a charge succeeds or a refund completes, you need absolute certainty that every system in your ecosystem gets that signal—without duplication, without loss, without timing surprises. The outbox pattern for payment events: a step-by-step guide gives you exactly that guarantee. We'll walk through how to write state and event atomically, relay those events to your message broker, and keep consumers idempotent so phantom events never corrupt your data.

Write state and event in one transaction

The core insight is that your application state and your event log must change together. When you record a payment as succeeded, you simultaneously insert a row into an outbox table. Both operations happen in the same database transaction—commit together or fail together. There is no window where state exists but the event doesn't, and vice versa.

Consider a typical payment capture flow:

async function capturePayment(chargeId: string, amount: number) {
  return await db.transaction(async (trx) => {
    // 1. Update payment state
    const payment = await trx('payments')
      .where({ id: chargeId })
      .update({
        status: 'captured',
        captured_amount: amount,
        captured_at: new Date(),
      })
      .returning('*');

    // 2. Insert outbox event (same transaction)
    await trx('outbox_events').insert({
      id: ulid(),
      aggregate_id: chargeId,
      aggregate_type: 'Payment',
      event_type: 'PaymentCaptured',
      event_data: JSON.stringify({
        chargeId,
        amount,
        capturedAt: payment.captured_at,
      }),
      created_at: new Date(),
      published: false,
    });

    return payment;
  });
}

The database enforces atomicity. If your application crashes mid-transaction, the database rolls back both the state change and the event insert. When the process restarts, the charge isn't marked as captured, so you'll retry the capture (idempotent at the API level). The outbox row never appears if it shouldn't.

Schema-wise, keep your outbox table simple:

Index on published = false and created_at so the relay process can efficiently scan unpublished events in creation order.

Relay publishes to your message broker

A background relay process polls the outbox table for unpublished events and delivers them to your message broker (Kafka, RabbitMQ, SQS, etc.). This decouples your payment capture from the fan-out to consumers.

The relay should run continuously and tolerate failures gracefully:

async function relayOutboxEvents() {
  const batchSize = 100;
  let hasMore = true;

  while (hasMore) {
    const events = await db('outbox_events')
      .where({ published: false })
      .orderBy('created_at', 'asc')
      .limit(batchSize);

    if (events.length === 0) {
      hasMore = false;
      break;
    }

    // Publish all events in this batch
    for (const event of events) {
      try {
        await publishToKafka(
          `payment-events.${event.event_type}`,
          {
            id: event.id,
            aggregateId: event.aggregate_id,
            type: event.event_type,
            data: JSON.parse(event.event_data),
            timestamp: event.created_at,
          }
        );

        // Mark as published only after successful publish
        await db('outbox_events')
          .where({ id: event.id })
          .update({
            published: true,
            published_at: new Date(),
          });
      } catch (error) {
        logger.error(`Failed to publish event ${event.id}:`, error);
        // Don't update published flag; retry on next iteration
        break; // Stop this batch, will retry next poll
      }
    }

    // Small delay before next batch
    await sleep(100);
  }
}

// Run continuously
setInterval(() => {
  relayOutboxEvents().catch(logger.error);
}, 5000); // Poll every 5 seconds

The key principle: mark the event as published only after the message broker acknowledges delivery. If the broker accepts your message and returns success, we update the database. If the relay crashes, the next restart will see unpublished events and retry.

Consumers stay idempotent

Because events can be replayed (after a relay crash, or if a consumer processes the same batch twice), every consumer handler must be idempotent. Processing the same PaymentCaptured event twice should leave the system in the same state as processing it once.

The easiest pattern is to store event IDs in a processed ledger:

async function handlePaymentCaptured(event: PaymentCapturedEvent) {
  // 1. Check if we've already processed this event
  const existing = await db('processed_events')
    .where({
      event_id: event.id,
      consumer_name: 'BillingService',
    });

  if (existing.length > 0) {
    logger.info(`Event ${event.id} already processed, skipping`);
    return;
  }

  // 2. Process the event (e.g., create invoice, send email)
  try {
    await createInvoiceForPayment(event.data.chargeId, event.data.amount);
    await notifyCustomer(event.data.chargeId, 'payment_captured');

    // 3. Record that we processed it
    await db('processed_events').insert({
      event_id: event.id,
      consumer_name: 'BillingService',
      processed_at: new Date(),
    });
  } catch (error) {
    logger.error(`Failed to handle event ${event.id}:`, error);
    throw error; // Message broker will retry
  }
}

Store processed_events in the same database as your application state. That way, recording that you processed an event and updating your state can happen in the same transaction, doubling down on atomicity.

Alternatively, if your consumer writes to a separate system (a data warehouse, a reporting database), use the event ID as an idempotency key in that system. Most modern databases and APIs support "insert if not exists" patterns that make this natural.

No lost or phantom events

The outbox pattern eliminates two failure modes:

Lost events: Without an outbox, you might publish directly to your broker and then crash before persisting state. Or vice versa. The outbox ensures that once state is written, the event will eventually reach consumers—retries are built in.

Phantom events: If you're not careful about consumer idempotency, a replay or network hiccup can cause the same business logic to run twice. Duplicate invoices, double refunds, angry customers. Idempotency keys and processed ledgers prevent that.

Let's trace a realistic scenario: you're processing a refund.

1. Refund API call comes in; you start a transaction.
2. You call Stripe's refund endpoint, get a success response.
3. You update payments.status = 'refunded' and insert into outbox.
4. Transaction commits. Your process crashes.
5. On restart, the relay finds the unpublished event and publishes it to Kafka.
6. The accounting consumer processes it; it checks its ledger and sees the event wasn't processed yet.
7. It creates a credit memo, records the event in its ledger.
8. Later, the relay crashes and restarts; it publishes the same event again.
9. The consumer checks its ledger, sees the event ID, skips processing, logs a no-op.

From the consumer's perspective, the event arrived exactly once. No double-booking, no phantom credits.

Common mistakes

• Publishing before updating state: If you publish to the broker and then update your database, a crash in between leaves you with a published event but no state change. Consumers might try to look up a payment that doesn't exist yet. Always update state and write to outbox in the same transaction.
• Marking events published too early: Don't mark an event as published until the broker acknowledges. If you set published = true before the publish call returns, a network timeout or broker failure means the event was never actually sent, but the relay won't retry it.
• Forgetting to index the outbox: Without an index on published = false and created_at, your relay's scan becomes a full table scan every 5 seconds. On a system with millions of old events, this tanks performance. Index aggressively.
• Consumer logic that isn't truly idempotent: "Check if processed, then process" only works if both steps are atomic. If you're calling out to an external service, make sure that service supports idempotency keys, or your processed ledger is written in the same transaction as the side effect (which is hard for external calls). Test replays explicitly.
• Relying on message ordering without the infrastructure: Kafka partitioning guarantees order within a partition, but if you publish all PaymentCaptured events to the same topic without partitioning by charge ID, they might be consumed out of order. Make sure your consumers can handle out-of-order events or partition carefully.

Key takeaways

• Write payment state and outbox event in a single database transaction—atomicity is your foundation.
• Run a relay process that polls unpublished events and publishes them to your message broker; mark published only after the broker confirms.
• Make consumers idempotent by storing processed event IDs or using idempotency keys; a replayed event should be a no-op.
• Index the outbox table on published and created_at to keep the relay efficient.
• Test your relay and consumer logic under failure scenarios—kill the relay mid-batch, replay events, verify no duplicates or losses.