Introduction
You've just shipped an order to your Kafka topic, but the broker is down for maintenance. Your code catches the exception, logs it, and moves on—meanwhile, your order sits in the database with no event record, orphaned and invisible to downstream services. This is the dual-write problem, and it's one of the most common sources of silent data loss in distributed systems. The outbox pattern in .NET: patterns, pitfalls and fixes gives us a reliable way to guarantee that events reach your broker, even when things go wrong.
Over the next few sections, we'll walk through why naive event publishing fails, how the outbox pattern solves it, how to implement it in ASP.NET Core, and where teams typically stumble when they try to put it into production.
The dual-write problem
The simplest approach to event-driven architecture looks innocent enough:
var order = new Order { CustomerId = customerId, Total = total };
await db.Orders.AddAsync(order, ct);
await db.SaveChangesAsync(ct);
// Publish to Kafka
await kafkaProducer.SendAsync(new OrderPlaced(order.Id), ct);
The problem surfaces immediately: what happens if Kafka is unreachable after the database commit? The order exists in your system, but no downstream service—your fulfillment warehouse, your analytics pipeline, your notification engine—knows about it. You're left with inconsistency: state in the database but no event in the broker.
You might wrap it in a transaction with a retry loop:
await db.Orders.AddAsync(order, ct);
await db.SaveChangesAsync(ct);
int attempts = 0;
while (attempts < 3) {
try {
await kafkaProducer.SendAsync(new OrderPlaced(order.Id), ct);
break;
} catch (KafkaException ex) {
attempts++;
if (attempts == 3) throw;
await Task.Delay(1000, ct);
}
}
This buys you a little time, but it doesn't solve the core issue: after your HTTP handler returns a 200 to the client and the process crashes, that retry loop is gone. Kafka might come back online tomorrow, but your order's event will never be published.
The outbox pattern solves this by treating the database as the source of truth for events. You write the event to a local table in the same transaction as your order, then a background process reliably ships it out to your broker.
Save events in the same transaction
The first step is to create an outbox table that holds messages until they're safely delivered:
public class OutboxMessage {
public long Id { get; set; }
public string AggregateId { get; set; }
public string EventType { get; set; }
public string Payload { get; set; }
public DateTime CreatedAt { get; set; }
public DateTime? ProcessedAt { get; set; }
}
Now, when you create an order, you write both the order and the event in a single transaction:
using (var transaction = await db.Database.BeginTransactionAsync(ct)) {
var order = new Order { CustomerId = customerId, Total = total };
await db.Orders.AddAsync(order, ct);
var outboxMessage = new OutboxMessage {
AggregateId = order.Id.ToString(),
EventType = "OrderPlaced",
Payload = JsonSerializer.Serialize(new OrderPlaced(order.Id)),
CreatedAt = DateTime.UtcNow
};
await db.Outbox.AddAsync(outboxMessage, ct);
await db.SaveChangesAsync(ct);
await transaction.CommitAsync(ct);
}
Because both writes happen in the same database transaction, either both succeed or both roll back. You can't end up with an order but no event row. From that point forward, the database is the single source of truth: if the row is in the outbox table, the event must be published.
For cleaner code, wrap this logic in a helper or use domain events. Many teams create a small abstraction:
public static class OutboxExtensions {
public static void AddOutboxMessage(
this DbContext db,
string aggregateId,
object domainEvent) {
var outboxMessage = new OutboxMessage {
AggregateId = aggregateId,
EventType = domainEvent.GetType().Name,
Payload = JsonSerializer.Serialize(domainEvent),
CreatedAt = DateTime.UtcNow
};
db.Outbox.Add(outboxMessage);
}
}
Then in your command handler:
var order = new Order { CustomerId = customerId, Total = total };
await db.Orders.AddAsync(order, ct);
db.AddOutboxMessage(order.Id.ToString(), new OrderPlaced(order.Id));
await db.SaveChangesAsync(ct);
Relay in the background
Now you need a hosted service that polls the outbox table, publishes unsent messages, and marks them processed. This is where reliability becomes critical: the relay must tolerate database hiccups, broker timeouts, and process restarts.
public class OutboxRelay : BackgroundService {
private readonly IServiceProvider _serviceProvider;
private readonly IKafkaProducer _kafkaProducer;
private readonly ILogger _logger;
public OutboxRelay(
IServiceProvider serviceProvider,
IKafkaProducer kafkaProducer,
ILogger logger) {
_serviceProvider = serviceProvider;
_kafkaProducer = kafkaProducer;
_logger = logger;
}
protected override async Task ExecuteAsync(CancellationToken ct) {
while (!ct.IsCancellationRequested) {
try {
await PublishPendingMessages(ct);
await Task.Delay(TimeSpan.FromSeconds(5), ct);
} catch (Exception ex) {
_logger.LogError(ex, "Outbox relay failed");
await Task.Delay(TimeSpan.FromSeconds(10), ct);
}
}
}
private async Task PublishPendingMessages(CancellationToken ct) {
using (var scope = _serviceProvider.CreateScope()) {
var db = scope.ServiceProvider.GetRequiredService();
var messages = await db.Outbox
.Where(m => m.ProcessedAt == null)
.OrderBy(m => m.CreatedAt)
.Take(100)
.ToListAsync(ct);
foreach (var message in messages) {
try {
// Deserialize and publish
await _kafkaProducer.SendAsync(
topic: GetTopicForEventType(message.EventType),
key: message.AggregateId,
value: message.Payload,
ct: ct);
message.ProcessedAt = DateTime.UtcNow;
await db.SaveChangesAsync(ct);
_logger.LogInformation(
"Published outbox message {MessageId} ({EventType})",
message.Id, message.EventType);
} catch (Exception ex) {
_logger.LogWarning(
ex,
"Failed to publish outbox message {MessageId}, will retry",
message.Id);
// Don't mark as processed; let it retry next cycle
}
}
}
}
private string GetTopicForEventType(string eventType) {
return eventType switch {
"OrderPlaced" => "orders.placed",
"OrderShipped" => "orders.shipped",
_ => throw new ArgumentException($"Unknown event type: {eventType}")
};
}
}
Register it in Program.cs:
services.AddHostedService();
The relay processes messages in small batches (100 at a time) to avoid memory bloat and provides predictable latency. If publishing fails, it logs a warning and leaves the row untouched, so it retries on the next cycle. This approach is simple and reliable: no fancy orchestration, just a steady background worker.
Consumers must be idempotent
With at-least-once delivery semantics, messages can be published multiple times. Your relay might publish an order event, crash before marking it processed, restart, and publish it again. Or Kafka itself might de-duplicate poorly, or your consumer might reprocess after a restart. Downstream services must handle duplicates gracefully.
The standard approach is to track processed message IDs:
public class ProcessedEvent {
public long Id { get; set; }
public long OutboxMessageId { get; set; }
public string EventType { get; set; }
public DateTime ProcessedAt { get; set; }
}
Before processing an event in your consumer, check if you've already seen it:
public class OrderPlacedConsumer : IConsumer {
private readonly ApplicationDbContext _db;
public async Task Consume(ConsumeContext context) {
var outboxId = context.Headers.Get("outbox-message-id");
// Check if we've already processed this message
var alreadyProcessed = await _db.ProcessedEvents
.AnyAsync(e => e.OutboxMessageId == outboxId);
if (alreadyProcessed) {
return; // Skip duplicate
}
// Process the event
var order = new ConfirmedOrder {
OrderId = context.Message.OrderId,
ConfirmedAt = DateTime.UtcNow
};
await _db.ConfirmedOrders.AddAsync(order);
// Record that we've processed this message
var processedEvent = new ProcessedEvent {
OutboxMessageId = outboxId,
EventType = "OrderPlaced",
ProcessedAt = DateTime.UtcNow
};
await _db.ProcessedEvents.AddAsync(processedEvent);
await _db.SaveChangesAsync();
}
}
Pass the outbox message ID in the Kafka message headers when the relay publishes:
var headers = new Headers {
{ "outbox-message-id", BitConverter.GetBytes(message.Id) }
};
await kafkaProducer.ProduceAsync(
new TopicPartition(topic, Partition.Any),
new Message {
Key = message.AggregateId,
Value = message.Payload,
Headers = headers
},
ct);
Alternatively, embed the message ID in the event payload itself. The key is consistency: choose one approach and stick with it across all consumers.
Common mistakes
- Not batching outbox reads: Querying one message at a time will tank your database under load. Always use
Take(N)and process in batches. - Forgetting to handle relay crashes: If the relay process dies after publishing but before marking the message processed, that's okay—it will retry. But you must tolerate duplicates in consumers.
- Publishing to the wrong topic: Make sure your outbox relay routes events to the correct Kafka topic based on event type. A mapping bug here can cause events to disappear invisibly.
- No monitoring: If your outbox table grows unbounded, your relay is likely failing silently. Track queue depth and alert on messages older than a threshold (e.g., 1 minute).
- Mixing transactional and non-transactional writes: If you write the outbox message but forget to include it in the same
SaveChangesAsynccall as your main aggregate, you've lost the guarantee. - Deleting old messages too aggressively: Some teams delete processed messages immediately. Instead, archive them for at least 7 days so you can replay events if needed or debug issues.
Key takeaways
- The dual-write problem—saving to the database then publishing to a broker—is inherently unreliable. One will always succeed while the other fails.
- The outbox pattern solves this by writing events to a local table in the same transaction as your business data, making the database the source of truth for events.
- A background hosted service polls the outbox table, publishes unsent messages to your broker, and marks them processed—using a simple, resilient loop rather than complex orchestration.
- Consumers must be idempotent because at-least-once delivery semantics guarantee duplicates will occur. Track processed message IDs to skip re-processing.
- Common pitfalls include forgetting to batch reads, not tolerating relay crashes, routing events to the wrong topics, and deleting old messages too quickly.
- Monitor outbox queue depth in production—growing backlogs indicate the relay is failing silently.
Comments
Loading comments…