The challenge of distributed reliability
You've just updated a user's subscription in your database. Now you need to notify the billing service, trigger an email, and log an analytics event—but your message queue goes down for 30 seconds. Which notifications get lost? How do you prevent duplicate charges or missed confirmations?
The outbox pattern is a design patterns guide for developers who need reliable event publishing from your database. Instead of writing data and hoping the event system catches everything, you write both the data change and the event into the same database transaction. A separate relay service then picks up unpublished events and delivers them to your event broker. This approach guarantees that if your database commit succeeds, the event will eventually be published—even if external systems fail.
Write data and event in one transaction
The core idea is elegant: when you persist a domain model change, you simultaneously insert an event record into an outbox table in the same transaction. If the transaction commits, both the data and the event are durable. If it rolls back, neither exists.
Let's say you're building an order management system in NestJS with TypeORM:
@Entity('orders')
export class Order {
@PrimaryGeneratedColumn('uuid')
id: string;
@Column()
customerId: string;
@Column()
totalAmount: number;
@Column()
status: string;
@CreateDateColumn()
createdAt: Date;
}
@Entity('outbox_events')
export class OutboxEvent {
@PrimaryGeneratedColumn('uuid')
id: string;
@Column()
aggregateId: string;
@Column()
eventType: string;
@Column('jsonb')
payload: Record;
@Column({ default: false })
published: boolean;
@CreateDateColumn()
createdAt: Date;
}
@Injectable()
export class OrderService {
constructor(
private dataSource: DataSource,
private orderRepository: Repository<Order>,
private outboxRepository: Repository<OutboxEvent>,
) {}
async createOrder(customerId: string, totalAmount: number): Promise<Order> {
const queryRunner = this.dataSource.createQueryRunner();
await queryRunner.connect();
await queryRunner.startTransaction();
try {
const order = queryRunner.manager.create(Order, {
customerId,
totalAmount,
status: 'pending',
});
const savedOrder = await queryRunner.manager.save(order);
const event = queryRunner.manager.create(OutboxEvent, {
aggregateId: savedOrder.id,
eventType: 'order.created',
payload: {
orderId: savedOrder.id,
customerId,
totalAmount,
createdAt: savedOrder.createdAt,
},
published: false,
});
await queryRunner.manager.save(event);
await queryRunner.commitTransaction();
return savedOrder;
} catch (error) {
await queryRunner.rollbackTransaction();
throw error;
} finally {
await queryRunner.release();
}
}
}
Both the order and the event are written in a single transaction. Your database enforces atomicity: either both succeed or both fail. There's no window where the order exists but the event doesn't.
A relay publishes events later
Now we need a separate process—the relay—that polls the outbox table for unpublished events and sends them to your event broker (Kafka, RabbitMQ, AWS SQS, etc.). This relay can run in a separate service or as a scheduled background job.
The relay pattern is simple but powerful:
- Query unpublished events from the outbox table.
- Publish them to the event broker.
- Mark them as published in the database.
- Repeat on a schedule (every second, every 100ms, etc.).
Here's a relay implementation using a scheduled NestJS service:
import { Injectable, Logger } from '@nestjs/common';
import { Cron } from '@nestjs/schedule';
import { Repository } from 'typeorm';
import { InjectRepository } from '@nestjs/typeorm';
import { ClientKafka } from '@nestjs/microservices';
@Injectable()
export class OutboxRelayService {
private readonly logger = new Logger(OutboxRelayService.name);
constructor(
@InjectRepository(OutboxEvent)
private outboxRepository: Repository<OutboxEvent>,
private kafkaClient: ClientKafka,
) {}
@Cron('*/1 * * * * *') // Every second
async relayEvents(): Promise<void> {
const unpublishedEvents = await this.outboxRepository.find({
where: { published: false },
order: { createdAt: 'ASC' },
take: 100, // Batch size
});
if (unpublishedEvents.length === 0) {
return;
}
for (const event of unpublishedEvents) {
try {
await this.kafkaClient
.emit(event.eventType, event.payload)
.toPromise();
event.published = true;
await this.outboxRepository.save(event);
this.logger.debug(
`Published event ${event.id} of type ${event.eventType}`,
);
} catch (error) {
this.logger.error(
`Failed to publish event ${event.id}: ${error.message}`,
);
// Do not mark as published; relay will retry on next cycle
}
}
}
}
The relay runs on a schedule and processes events in batches. If publishing fails, the event remains unpublished and will be retried on the next cycle. This decouples the database write from the event publication latency.
At-least-once delivery
The outbox pattern guarantees at-least-once delivery, not exactly-once. This means an event might be published more than once, especially if the relay crashes after sending the event to Kafka but before marking it as published in the database.
Consider a failure scenario:
- Relay publishes event to Kafka.
- Kafka broker confirms receipt.
- Relay's connection to the database drops before it can mark the event as published.
- On restart, the relay sees the same event as unpublished and sends it again.
This is actually acceptable because we design our consumers to handle duplicates through idempotency. At-least-once delivery paired with idempotent consumers is a pragmatic choice that avoids the complexity and performance cost of distributed consensus algorithms.
To improve reliability further, you can track delivery attempts and implement backoff strategies:
@Entity('outbox_events')
export class OutboxEvent {
@PrimaryGeneratedColumn('uuid')
id: string;
@Column()
aggregateId: string;
@Column()
eventType: string;
@Column('jsonb')
payload: Record<string, unknown>;
@Column({ default: false })
published: boolean;
@Column({ default: 0 })
attemptCount: number;
@Column({ nullable: true })
lastAttemptAt: Date;
@Column({ nullable: true })
lastError: string;
@CreateDateColumn()
createdAt: Date;
}
// In the relay service:
async relayEvents(): Promise<void> {
const unpublishedEvents = await this.outboxRepository.find({
where: [
{ published: false, attemptCount: 0 },
{ published: false, lastAttemptAt: LessThan(new Date(Date.now() - 5000)) }, // Retry after 5s
],
order: { createdAt: 'ASC' },
take: 100,
});
for (const event of unpublishedEvents) {
if (event.attemptCount >= 10) {
this.logger.warn(`Event ${event.id} exceeded max retries`);
continue;
}
try {
await this.kafkaClient.emit(event.eventType, event.payload).toPromise();
event.published = true;
await this.outboxRepository.save(event);
} catch (error) {
event.attemptCount += 1;
event.lastAttemptAt = new Date();
event.lastError = error.message;
await this.outboxRepository.save(event);
}
}
}
Now the relay respects backoff windows and tracks how many times it has tried to publish an event. Events that fail repeatedly are logged for manual investigation rather than hammering your event broker.
Consumers must be idempotent
Because at-least-once delivery means events can arrive more than once, every consumer must be idempotent. Processing the same event twice should produce the same result as processing it once.
The standard approach is to track an event ID or correlation ID and skip processing if you've already seen it:
@Injectable()
export class BillingConsumer {
constructor(
private billingRepository: Repository<BillingRecord>,
private dataSource: DataSource,
) {}
async handleOrderCreated(payload: any): Promise<void> {
const { orderId, customerId, totalAmount } = payload;
const eventId = payload.eventId || `order.created.${orderId}`;
// Check if we've already processed this event
const existing = await this.billingRepository.findOne({
where: { eventId },
});
if (existing) {
// Already processed; skip
return;
}
const queryRunner = this.dataSource.createQueryRunner();
await queryRunner.connect();
await queryRunner.startTransaction();
try {
const billingRecord = queryRunner.manager.create(BillingRecord, {
eventId,
orderId,
customerId,
amount: totalAmount,
status: 'pending',
});
await queryRunner.manager.save(billingRecord);
// Publish any events from this aggregate
const event = queryRunner.manager.create(OutboxEvent, {
aggregateId: orderId,
eventType: 'billing.initiated',
payload: { orderId, customerId, totalAmount },
published: false,
});
await queryRunner.manager.save(event);
await queryRunner.commitTransaction();
} catch (error) {
await queryRunner.rollbackTransaction();
throw error;
} finally {
await queryRunner.release();
}
}
}
The consumer stores a unique event ID with each processed record. On duplicate arrival, it detects the ID already exists and skips processing. This is idempotency through the database: the operation is safe to repeat.
Common mistakes
- Publishing outside the transaction: Inserting into the outbox inside the transaction but publishing to Kafka outside of it defeats the whole pattern. Keep the outbox write atomic with your domain write.
- Forgetting to query only unpublished events: If your relay doesn't filter by
published = false, it will continuously re-process the same events and waste resources. - No dead-letter handling: Events that fail repeatedly need a way to surface alerts. Without tracking attempts and errors, failed events silently disappear.
- Consumers not tracking event IDs: If consumers don't implement idempotency checks, duplicate events will create duplicate state. Use correlation IDs, event IDs, or aggregate IDs to detect duplicates.
- Tight relay polling intervals: Polling every millisecond puts pressure on your database. A 100–500ms interval usually balances latency and throughput well for most use cases.
Key takeaways
- The outbox pattern ensures that writing your domain data and publishing an event happen atomically in a single database transaction.
- A relay service polls the outbox table on a schedule and publishes events to your event broker, decoupling the write latency from event delivery.
- At-least-once delivery is acceptable when paired with idempotent consumers; it avoids the complexity of exactly-once semantics.
- Design every consumer to detect and skip duplicate events using event IDs or correlation IDs.
- Track retry attempts and errors in the outbox table to catch and alert on events that fail repeatedly.
- The outbox pattern is particularly valuable in e-commerce, billing, and transactional workflows where event loss or duplication has business impact.
Comments
Loading comments…