Real-time market data fan-out: best practices explained

Real-time market data fan-out: best practices explained

When you're building a trading platform, pushing price updates to hundreds of connected clients is straightforward in theory—subscribe to a Kafka topic, broadcast everything to everyone. In practice, you'll quickly hit memory limits, CPU saturation, and unhappy traders staring at frozen tickers. We've learned through production outages that real-time market data fan-out demands a deliberate architecture: selective subscriptions, smart conflation, stateful snapshots, and efficient rendering. This article distills the patterns that kept our trading platform responsive during peak volatility.

Subscribe per symbol — Clients subscribe only to the symbols they display

The first instinct many teams have is to broadcast all market data to all connected clients and let JavaScript filter what matters. This works until you have 500 simultaneous traders watching different instruments across multiple exchanges. Your server is now pushing data for every single symbol to every connection, even though a client only cares about 10 tickers.

Instead, implement per-symbol subscriptions. Clients declare what they need, and your WebSocket server maintains a subscription map. When market data arrives on Kafka, you fan out only to clients actively watching that symbol.

// Server-side subscription management
interface ClientSubscription {
  clientId: string;
  symbols: Set<string>;
}

const subscriptions = new Map<string, Set<string>>(); // symbol -> Set of clientIds

// Client subscribes to specific symbols
ws.on('message', (msg) => {
  const { type, symbols } = JSON.parse(msg);
  
  if (type === 'subscribe') {
    subscriptions.set(clientId, new Set(symbols));
  }
});

// On incoming Kafka message
kafkaConsumer.on('message', (message) => {
  const { symbol, price, timestamp } = JSON.parse(message.value);
  
  // Only broadcast to clients subscribed to this symbol
  const interestedClients = subscriptions.get(symbol) || new Set();
  interestedClients.forEach((cid) => {
    if (clients[cid]?.readyState === WebSocket.OPEN) {
      clients[cid].send(JSON.stringify({ symbol, price, timestamp }));
    }
  });
});

This approach reduces network traffic and CPU usage on the server dramatically. A trader watching 15 symbols no longer receives updates for the other 985 instruments your platform trades. You'll also want to clean up subscriptions when clients disconnect to prevent memory leaks.

Conflate updates — Send the latest price per interval instead of every tick

Market data flows fast. On a busy exchange, you might receive 50 price updates per second for a single symbol. Broadcasting all 50 to all interested clients overwhelms the WebSocket pipeline and the browser's event loop. Users don't need every single tick; they need timely, current prices.

Conflation is the practice of collapsing multiple rapid updates into a single message per interval. Instead of sending price #1, then #2, then #3, you buffer updates and send only the latest state every 100ms or 500ms, depending on your latency requirements.

// Conflation buffer per symbol
const conflationBuffers = new Map<string, any>();
const CONFLATION_INTERVAL_MS = 100;

kafkaConsumer.on('message', (message) => {
  const update = JSON.parse(message.value);
  const { symbol } = update;
  
  // Store the latest update for this symbol
  conflationBuffers.set(symbol, update);
});

// Flush conflation buffers periodically
setInterval(() => {
  conflationBuffers.forEach((update, symbol) => {
    const interestedClients = subscriptions.get(symbol) || new Set();
    interestedClients.forEach((clientId) => {
      if (clients[clientId]?.readyState === WebSocket.OPEN) {
        clients[clientId].send(JSON.stringify(update));
      }
    });
  });
  
  // Clear buffer after flush
  conflationBuffers.clear();
}, CONFLATION_INTERVAL_MS);

The trade-off is minimal latency increase (100–500ms is imperceptible in trading UIs) for a dramatic reduction in message volume. We measured a 60–70% reduction in outbound WebSocket traffic by conflating at 100ms intervals. Your bandwidth costs and server load both improve, and clients handle fewer DOM updates.

Snapshot on subscribe — New subscribers get the current state, then deltas

A client opens a trading dashboard and subscribes to ten symbols. If you only send future updates, their prices stay blank until the next tick arrives. The user sees an incomplete, potentially stale interface for seconds. Instead, provide a snapshot: when a client subscribes, immediately send the latest known state for each symbol, then stream incremental updates.

Kafka's log retention and consumer group mechanics make this pattern natural if you structure your data correctly. Maintain a separate state store (Redis or in-memory) of the latest price per symbol, and query it on subscription.

// In-memory state store (or Redis in production)
const latestPrices = new Map<string, any>();

// Update state whenever Kafka data arrives
kafkaConsumer.on('message', (message) => {
  const update = JSON.parse(message.value);
  latestPrices.set(update.symbol, update);
  // ... conflation logic ...
});

// On client subscription
ws.on('message', (msg) => {
  const { type, symbols } = JSON.parse(msg);
  
  if (type === 'subscribe') {
    // Send current snapshot
    const snapshot = symbols.map((sym) => {
      const latest = latestPrices.get(sym);
      return latest || { symbol: sym, price: null, timestamp: null };
    });
    
    ws.send(JSON.stringify({
      type: 'snapshot',
      data: snapshot,
    }));
    
    // Track subscription for future deltas
    subscriptions.set(clientId, new Set(symbols));
  }
});

This ensures zero visual lag on subscribe. Traders see prices immediately, even if no update for that symbol has arrived since their connection was established. The separation of snapshot and delta streams also makes your client code clearer: initial render uses snapshot data, stream listeners update individual cells.

Render efficiently — Batch DOM updates per animation frame

Even with conflation and selective subscriptions, a busy trading UI receives hundreds of WebSocket messages per second. If your client code updates the DOM synchronously on every message, the browser's render thread chokes. The solution is simple: batch updates using requestAnimationFrame to align with the browser's refresh cycle (typically 60 Hz, one frame every ~16ms).

// Client-side batching
let pendingUpdates = new Map<string, any>();
let rafScheduled = false;

ws.onmessage = (event) => {
  const { type, data } = JSON.parse(event.data);
  
  if (type === 'snapshot') {
    // Snapshot: update state immediately
    data.forEach((price) => {
      priceState[price.symbol] = price;
    });
    renderUI();
  } else {
    // Delta: queue for next frame
    const update = data; // { symbol, price, timestamp }
    pendingUpdates.set(update.symbol, update);
    
    if (!rafScheduled) {
      rafScheduled = true;
      requestAnimationFrame(flushUpdates);
    }
  }
};

function flushUpdates() {
  // Apply all pending updates at once
  pendingUpdates.forEach((update) => {
    priceState[update.symbol] = update;
    updateCellDOM(update.symbol, update.price);
  });
  
  pendingUpdates.clear();
  rafScheduled = false;
}

In React or Vue, this pattern integrates naturally with component state. Use a single state update (or a single batch of updates) per frame rather than triggering renders on every message. For vanilla JavaScript, maintain a pending-updates map and flush once per animation frame. The visual result is smooth, jank-free updates aligned with your monitor's refresh rate.

Common mistakes

• Broadcasting to all clients regardless of interest. Keeping this simple saves development time but costs you in scalability. By the time you need to optimize, architectural changes become painful. Build subscription filtering from day one.
• Conflating too aggressively or not at all. Too little conflation (sending every tick) wastes bandwidth. Too much (e.g., 5-second intervals) makes prices feel stale. Experiment with your user base; we landed on 100–200ms as a sweet spot for equities.
• Forgetting to handle subscription cleanup. Clients disconnect without sending explicit unsubscribe messages. Implement a heartbeat or monitor WebSocket readyState; remove stale subscriptions regularly or you'll leak memory.
• Rendering every update synchronously in client code. Even with a perfect backend, blocking the main thread on the client kills responsiveness. Always batch updates to requestAnimationFrame or your framework's render batching.
• Storing snapshots in the wrong place. Querying a database for every subscription is too slow. Keep a hot cache (in-memory or Redis) of the latest state. In-memory works for moderate symbol counts; use Redis for multi-server setups.

Key takeaways

• Implement per-symbol subscriptions so clients receive only data they need, reducing server bandwidth and CPU.
• Conflate rapid updates into a single message per 100–500ms interval, cutting message volume by 60–70% with negligible latency impact.
• Send a snapshot of current prices when clients subscribe, eliminating visual lag and ensuring a complete initial render.
• Batch DOM updates per animation frame on the client to prevent main-thread blocking and jank during high-frequency updates.
• Combine these four patterns to build a trading UI that scales from 50 concurrent users to 500+ without degrading latency or responsiveness.