We've all seen it: an event-driven system that looked elegant on the whiteboard but turned into a distributed debugging nightmare in production. After 5+ years building platforms with Kafka and WebSockets, I've watched teams fall into the same traps repeatedly. Let me walk you through the most painful event-driven architecture pitfalls and how to avoid them.
Events as Remote Calls — Waiting for Event Replies Recreates Synchronous Coupling
One of the biggest misconceptions is treating events like request-reply messages. You emit an event and then wait for a response, essentially recreating synchronous coupling across service boundaries. This defeats the entire purpose of event-driven architecture.
I saw this in a ride-hailing platform we built. The booking service would emit a ride.requested event and then immediately block, waiting for a response from the driver-matching service:
// ❌ WRONG: Synchronous coupling disguised as events
async function createRideRequest(userId: string, pickup: Location) {
const rideId = generateId();
// Emit event
await kafka.send({
topic: 'ride.requested',
messages: [{ value: JSON.stringify({ rideId, userId, pickup }) }]
});
// Now wait for a reply — this is blocking!
const matchResult = await waitForMatchResult(rideId, 5000);
if (!matchResult) {
throw new Error('Matching timeout');
}
return { rideId, driver: matchResult.driver };
}
What happens when the matching service is slow or down? The booking service times out. What if matching takes 30 seconds sometimes? Your API gateway hits its timeout. You're no longer resilient; you're just slower.
The correct approach is to decouple completely. Emit the event, persist the booking in a pending state, and let the matching service independently emit a ride.matched event that triggers the next step:
// ✅ CORRECT: True event-driven, fully decoupled
async function createRideRequest(userId: string, pickup: Location) {
const rideId = generateId();
// Persist in pending state
await db.rides.insert({
id: rideId,
userId,
pickup,
status: 'pending'
});
// Emit event and return immediately
await kafka.send({
topic: 'ride.requested',
messages: [{
value: JSON.stringify({ rideId, userId, pickup }),
key: rideId // Partition by ride for ordering
}]
});
// Return immediately; client polls or uses WebSocket for updates
return { rideId, status: 'pending' };
}
// Matching service independently consumes and responds
consumer.on('ride.requested', async (message) => {
const { rideId, userId, pickup } = JSON.parse(message.value);
const driver = await findNearestDriver(pickup);
await kafka.send({
topic: 'ride.matched',
messages: [{
value: JSON.stringify({ rideId, driverId: driver.id }),
key: rideId
}]
});
});
Now matching can take 5 seconds or 50 seconds; the booking service never cares. If matching fails, it emits a separate event. Services evolve independently. This is real event-driven architecture.
No Schemas — Silent Breaking Changes Across Teams
Events without schemas are a maintenance disaster waiting to happen. One team adds a field, another team removes one, and suddenly downstream consumers silently break because they're parsing optional fields as required.
I worked on a travel platform where the payments service emitted a payment.completed event. Initially it looked like this:
{
paymentId: "pay_123",
amount: 5000,
currency: "USD"
}
Six months later, payments added a fee field. The invoicing service expected it, but the legacy email service didn't know about it. Then someone removed the currency field "to save space," and the accounting service broke silently because it assumed currency was always there.
Use Apache Avro or JSON Schema. Make it mandatory. Here's a Kafka + Avro setup:
// Define schema using Avro
const paymentSchema = {
type: 'record',
namespace: 'com.platform.payments',
name: 'PaymentCompleted',
fields: [
{ name: 'paymentId', type: 'string' },
{ name: 'amount', type: 'long' },
{ name: 'currency', type: 'string' },
{ name: 'fee', type: ['null', 'long'], default: null },
{ name: 'timestamp', type: 'long' }
]
};
// Register with schema registry
const schemaRegistry = new SchemaRegistry({ host: 'http://localhost:8081' });
await schemaRegistry.register(paymentSchema);
// Produce with schema validation
const { id } = await schemaRegistry.getLatestSchemaId('payments');
await producer.send({
topic: 'payments',
messages: [{
value: await schemaRegistry.encode(id, {
paymentId: "pay_123",
amount: 5000,
currency: "USD",
fee: 100,
timestamp: Date.now()
})
}]
});
Schemas act as contracts. When someone tries to add a required field, the schema registry rejects it until all consumers agree. When you remove a field, you increment the schema version. Consumers can be updated on their own schedule, and you get proper versioning instead of silent failures.
No Idempotency — Duplicates Cause Double Charges and Double Emails
Kafka guarantees "at-least-once" delivery by default, not exactly-once. Network hiccups, consumer crashes, retries—they all mean events can be processed multiple times. If your handlers aren't idempotent, you'll charge customers twice or send the same email three times.
I've seen this destroy trust. A SaaS billing system processed the same subscription charge twice because the consumer crashed right after updating the database but before committing the offset. Customer support got overwhelmed.
Every event handler must be idempotent:
// ❌ WRONG: Not idempotent
async function handlePaymentCompleted(event: PaymentEvent) {
// If this crashes after insert but before offset commit,
// next retry will double-charge
await db.charges.insert({
paymentId: event.paymentId,
amount: event.amount,
customerId: event.customerId
});
await emailService.send({
to: event.customerEmail,
subject: 'Payment received'
});
}
// ✅ CORRECT: Idempotent using upsert
async function handlePaymentCompleted(event: PaymentEvent) {
// Use database constraint to make it idempotent
await db.charges.upsert(
{ paymentId: event.paymentId },
{
paymentId: event.paymentId,
amount: event.amount,
customerId: event.customerId,
processedAt: new Date()
}
);
// Email is also safe: idempotency key prevents duplicates
await emailService.send({
to: event.customerEmail,
subject: 'Payment received',
idempotencyKey: `charge_${event.paymentId}`
});
}
The key patterns are: use unique constraints (database naturally deduplicates), maintain an idempotency store (store processed message IDs), or use idempotency keys with your dependencies (payment providers, email services often support this). Never rely on "this probably won't happen twice."
No Observability — Nobody Knows Where an Event Got Stuck
Events flow through multiple services, transformations, and systems. Without proper observability, debugging is nightmare fuel. Where did the event go? Is it still processing? Did it get stuck in a dead-letter queue?
I worked on a CRM platform where orders would sometimes not sync to the fulfillment system. We had logs everywhere but no way to track one order's journey through five services. It took three days to find that a topic was misconfigured and events were being silently dropped.
Implement end-to-end tracing with correlation IDs:
// Generate or propagate correlation ID
function generateCorrelationId(): string {
return `trace_${Date.now()}_${Math.random().toString(36).substr(2, 9)}`;
}
// Producer: include trace context
async function emitEvent(event: any, correlationId: string) {
await kafka.send({
topic: event.topic,
messages: [{
key: event.aggregateId,
value: JSON.stringify({
...event,
_meta: {
correlationId,
timestamp: Date.now(),
source: 'orders-service'
}
})
}]
});
logger.info('Event emitted', {
correlationId,
eventType: event.type,
aggregateId: event.aggregateId
});
}
// Consumer: extract and use trace context
consumer.on('order.created', async (message) => {
const event = JSON.parse(message.value);
const { correlationId } = event._meta;
const childLogger = logger.child({ correlationId });
try {
childLogger.info('Processing event');
await processOrder(event);
childLogger.info('Event processed successfully');
} catch (error) {
childLogger.error('Event processing failed', { error });
// Send to dead-letter topic with trace info
await deadLetterQueue.send({
topic: 'dlq',
messages: [{
value: JSON.stringify({
event,
error: error.message,
correlationId
})
}]
});
}
});
Add monitoring dashboards: event lag per topic, processing latency, error rates, dead-letter queue depth. When something breaks, you can query by correlation ID and see the entire event's journey across all services within seconds instead of days.
Common Mistakes
- Synchronous waiting for events: Treating event consumption as RPC ruins your resilience.
- Unversioned events: No schema means silent breakage when fields change.
- Assuming exactly-once delivery: Kafka doesn't guarantee it; your code must handle duplicates.
- No trace context: Events with no correlation ID make debugging take days.
- Fire-and-forget without monitoring: Emitting events with zero visibility into whether they're actually processed.
- Processing events synchronously in handlers: Long-running operations should spawn async tasks, not block the consumer.
- Sharing topics across unrelated events: Different event types should have separate topics for clearer semantics and independent scaling.
Key Takeaways
- Emit events and return immediately; never wait for replies. Decouple by design.
- Use schemas (Avro, JSON Schema, Protobuf) and register them centrally. Make versioning explicit.
- Design every event handler to be idempotent using upserts, unique constraints, or idempotency keys.
- Add correlation IDs to all events and propagate them through logs and traces.
- Monitor event lag, processing latency, and dead-letter queue depth in real time.
- Test failure scenarios: slow consumers, network partitions, crashes mid-processing.
- Document event semantics and ownership; one team should own each event type.
Comments
Loading comments…