Bridging Kafka to WebSockets: what developers should know

You've built a solid Kafka pipeline that captures domain events across your platform, but your frontend still polls endpoints every few seconds for updates. That's where bridging Kafka to WebSockets changes everything: you turn asynchronous backend events into real-time UI updates that reach users the moment something happens. In this article, I'll walk you through the architecture, patterns, and gotchas I've learned building live-update systems at scale.

The Bridge — Event Flow from Kafka to Connected Clients

At its core, a Kafka-to-WebSockets bridge is a gateway service that sits between your event stream and your WebSocket connections. Here's the flow:

1. A domain service (order, payment, inventory) publishes an event to Kafka
2. The gateway consumes that topic in real time
3. The gateway maps the event to a user room or topic
4. The gateway pushes the transformed payload to all connected sockets in that room

I built this pattern first for a ride-hailing platform where driver location updates, ride status changes, and fare calculations all needed to hit passenger and driver apps instantly. Without a bridge, we'd have no way to correlate LocationUpdatedEvent messages in Kafka with specific WebSocket connections authenticated as driver ID 42.

Here's a simplified NestJS gateway that consumes a Kafka topic and pushes to rooms:

import { Injectable } from '@nestjs/common';
import { OnModuleInit } from '@nestjs/common';
import { Server } from 'socket.io';
import { KafkaService } from './kafka.service';

@Injectable()
export class OrderEventGateway implements OnModuleInit {
  private io: Server;

  constructor(private kafkaService: KafkaService) {}

  setServer(io: Server) {
    this.io = io;
  }

  async onModuleInit() {
    // Subscribe to order events from Kafka
    await this.kafkaService.subscribe('order-events', async (message) => {
      const event = JSON.parse(message.value.toString());
      
      // Map event to a user room (e.g., "user:123")
      const userId = event.userId;
      const room = `user:${userId}`;
      
      // Push to all sockets in that room
      this.io.to(room).emit('order:updated', {
        orderId: event.orderId,
        status: event.status,
        timestamp: event.timestamp,
      });
    });
  }
}

This is deceptively simple, but it solves the core problem: events flow through Kafka, and the gateway acts as the translator that connects them to live connections.

Filter Early — Push Only What Clients Need

Broadcasting every event to every connected socket will destroy your infrastructure. If you have 100,000 concurrent connections and push 1,000 events per second, even a 1 KB payload per event becomes 100 MB/s of network traffic. You need filters at the gateway level.

The key principle: filter in the consumer, not on the client. If an event doesn't apply to any connected user, don't even deserialize it fully.

Here's a filtering strategy I use:

async onModuleInit() {
  await this.kafkaService.subscribe('order-events', async (message) => {
    const event = JSON.parse(message.value.toString());
    
    // Early filter: only push if there are active connections for this user
    const room = `user:${event.userId}`;
    const socketsInRoom = await this.io.in(room).fetchSockets();
    
    if (socketsInRoom.length === 0) {
      // No one is listening, skip expensive processing
      return;
    }
    
    // Filter by event type: maybe we only push certain status changes
    if (!['ORDER_PLACED', 'ORDER_SHIPPED', 'ORDER_DELIVERED'].includes(event.type)) {
      return;
    }
    
    // Now transform and emit
    this.io.to(room).emit('order:updated', this.transformEvent(event));
  });
}

In practice, I've also pushed filters upstream into Kafka consumer groups. Use topic partitioning or Kafka Streams to pre-filter events so the gateway only sees relevant messages. For a trading platform, we consumed only trade fills for users who had active WebSocket connections—determined by querying a Redis cache of connected users updated in real time.

The discipline here prevents cascading failures. One noisy topic shouldn't spike CPU or memory on your gateway instances.

Transform for Clients — API Contracts Matter

Your internal domain events have fields optimized for logging, auditing, and service-to-service communication. Your frontend clients need a different shape entirely.

A PaymentProcessedEvent in Kafka might look like:

{
  eventId: "evt_123",
  eventType: "payment.processed",
  aggregateId: "order_456",
  aggregateVersion: 3,
  timestamp: "2025-01-15T10:23:45Z",
  metadata: {
    userId: "user_789",
    source: "stripe_webhook",
    idempotencyKey: "key_xyz"
  },
  payload: {
    transactionId: "txn_abc",
    amount: 9999,
    currency: "USD",
    fee: 299,
    net: 9700
  }
}

Your frontend doesn't care about eventId, aggregateVersion, or idempotencyKey. It cares about: "The payment succeeded, here's the total charged, here's the final amount after fees, show a confirmation." Your WebSocket payload should be lean and direct:

@Injectable()
export class PaymentTransformer {
  transform(event: PaymentProcessedEvent): WebSocketPayload {
    return {
      type: 'payment_complete',
      orderId: event.aggregateId,
      amount: event.payload.amount,
      displayAmount: this.formatCurrency(event.payload.amount, event.payload.currency),
      timestamp: event.timestamp,
    };
  }
}

Separation of concerns is critical here. Your event schema evolves independently of your WebSocket API. If you add a field to Kafka events for compliance tracking, clients don't break. If you deprecate a WebSocket field, Kafka events keep it for auditing.

I've seen teams skip this step and end up with their Kafka event schema leaking into frontend code. Three months later, you're afraid to add a field to internal events because somewhere a React component depends on it. Transformation layers prevent that coupling.

Stateless Gateways — Keep Connection State Minimal

Your gateway instances should be cattle, not pets. If one crashes, a new one spins up and connections migrate to it. That means minimizing in-memory state.

The temptation is to store event processing state in the gateway:

// Don't do this (bad pattern)
private eventCache = new Map();

async onModuleInit() {
  await this.kafkaService.subscribe('order-events', async (message) => {
    const event = JSON.parse(message.value.toString());
    this.eventCache.set(event.orderId, event); // ← Tied to this instance
    // ...
  });
}

If this gateway crashes, all that cached state is lost. New events arrive but have no context. In a trading system, losing recent price-update state could mean sending stale data to clients.

Instead, use Redis as your shared state layer:

constructor(
  private kafkaService: KafkaService,
  private redis: RedisService
) {}

async onModuleInit() {
  await this.kafkaService.subscribe('order-events', async (message) => {
    const event = JSON.parse(message.value.toString());
    const room = `user:${event.userId}`;
    
    // Persist state to Redis (survives gateway restart)
    await this.redis.set(
      `order:${event.orderId}:latest`,
      JSON.stringify(event),
      'EX',
      3600 // 1 hour TTL
    );
    
    // Push to connected clients
    this.io.to(room).emit('order:updated', this.transformEvent(event));
  });
}

// When a client reconnects, fetch latest state from Redis
async handleClientConnect(socket: Socket, userId: string) {
  const room = `user:${userId}`;
  socket.join(room);
  
  // Hydrate with latest state
  const keys = await this.redis.keys(`order:*:latest`);
  for (const key of keys) {
    const event = await this.redis.get(key);
    if (event) {
      socket.emit('order:hydrate', JSON.parse(event));
    }
  }
}

Now your gateway is truly stateless. Kafka is the source of truth, Redis holds transient state for quick hydration on reconnect, and the gateway is just a fan-out layer. Any instance can handle any connection.

For a SaaS platform with millions of users, this architecture scales horizontally. Add more gateway instances behind a load balancer, and they all consume the same Kafka topics independently. Connection state is distributed via Redis. The only bottleneck is your Kafka cluster, which you've already optimized.

Common Mistakes

• Blocking the Kafka consumer thread. If transforming or emitting is slow, your consumer lags and events pile up in Kafka. Use async/await and emit fire-and-forget where safe, or use a work queue inside the gateway to decouple consumption from emission.
• No error handling on socket emit. If a socket is closed or full (backpressure), emitting throws. Wrap emits in try-catch and track failed sends per socket, then reconnect or drop the client gracefully.
• Flooding with redundant events. If multiple services publish overlapping events, clients see duplicates. Deduplicate at the gateway using event ID + short-lived Redis cache before emitting.
• Losing events between Kafka offset and socket emission. Commit your Kafka offset only after successfully emitting (or queuing to emit). If you commit early, a crash loses those events. If you commit late, you reprocess on restart—that's fine, clients handle it.
• Not monitoring gateway lag. Track how far behind your consumer is. If it's lagging 10 seconds, clients are receiving stale events and don't know it. Alert on consumer lag.

Key Takeaways

• A Kafka-to-WebSockets bridge consumes from Kafka, maps events to user rooms, and pushes transformed payloads to connected sockets—solving the latency problem of polling.
• Filter events at the gateway level before deserializing or emitting; broadcast nothing to no one.
• Transform internal domain events into lean, client-friendly payloads; keep your event schema and WebSocket API decoupled.
• Store durable state in Redis, not in gateway memory; keep instances stateless so they're replaceable and scalable.
• Monitor consumer lag, handle socket errors, deduplicate, and commit offsets safely—these operational details prevent silent failures in production.