Payload size matters: a practical Kafka & WebSockets guide

Payload size matters

I learned this lesson the hard way during my second year at a ride-hailing platform: we had built a real-time driver availability system using WebSockets and Kafka, pushing driver location updates to thousands of subscribers. Within weeks, our AWS data transfer costs tripled, our p99 latency spiked, and mobile clients started dropping connections. The root cause wasn't complex logic—it was that every byte we sent went to every single subscriber, and we were broadcasting full driver objects with unnecessary fields. Payload size matters because in a fan-out architecture, small inefficiencies compound across every connection and every message.

Today, I want to walk through practical strategies I use to optimize payload sizes when building real-time systems. These aren't theoretical exercises; they're decisions that directly impact your infrastructure costs, network utilization, and user experience.

Shorten keys or use binary encoding

The simplest wins come from reconsidering how you structure your messages. When a driver location update goes to 5,000 subscribers, every character in your JSON keys is replicated 5,000 times. If your message looks like this:

interface DriverLocation {
  driver_id: string;
  latitude: number;
  longitude: number;
  vehicle_type: string;
  is_available: boolean;
  current_speed: number;
  heading_direction: number;
  last_updated_timestamp: string;
  passenger_count: number;
  ride_status: string;
}

const update = {
  driver_id: "d_12345",
  latitude: 6.9271,
  longitude: 80.7789,
  vehicle_type: "tuk",
  is_available: true,
  current_speed: 25,
  heading_direction: 180,
  last_updated_timestamp: "2024-01-15T14:32:00Z",
  passenger_count: 0,
  ride_status: "idle"
};

That's roughly 450 bytes just for one update. Over a WebSocket connection handling 20 updates per second, you're looking at 9 KB/s per connection. Across 5,000 subscribers, that's 45 MB/s leaving your server.

I switch to abbreviated keys immediately when I identify high-frequency, high-fanout scenarios:

const update = {
  d: "d_12345",
  lt: 6.9271,
  ln: 80.7789,
  vt: "tuk",
  av: true,
  sp: 25,
  hd: 180,
  ts: 1705335120000,
  pc: 0,
  rs: "idle"
};

That same update shrinks to about 140 bytes. The client maintains a schema, so it knows that d means driver_id and lt means latitude. If you want to go further, consider binary encoding via MessagePack or Protocol Buffers. With MessagePack, that same update becomes roughly 80 bytes:

import { pack } from "msgpackr";

const compressed = pack({
  d: "d_12345",
  lt: 6.9271,
  ln: 80.7789,
  vt: "tuk",
  av: true,
  sp: 25,
  hd: 180,
  ts: 1705335120000,
  pc: 0,
  rs: "idle"
});

console.log(compressed.length); // ~80 bytes

The trade-off is client-side deserialization overhead and debugging difficulty. At a ride-hailing scale, the bandwidth savings justify it. For smaller systems, abbreviated JSON keys strike the right balance.

Send deltas instead of full objects

Full object updates are wasteful. If a driver's location updates every second but their vehicle type changes once per week, you're transmitting the same vehicle type 604,800 times unnecessarily.

I implement delta encoding by tracking state server-side and only broadcasting fields that changed:

class DriverLocationBroadcaster {
  private previousState = new Map();

  computeDelta(driverId: string, current: DriverLocation) {
    const previous = this.previousState.get(driverId) || {};
    const delta: Record = { d: driverId };

    if (current.latitude !== previous.latitude) delta.lt = current.latitude;
    if (current.longitude !== previous.longitude) delta.ln = current.longitude;
    if (current.speed !== previous.speed) delta.sp = current.speed;
    if (current.heading !== previous.heading) delta.hd = current.heading;
    if (current.isAvailable !== previous.isAvailable) delta.av = current.isAvailable;

    this.previousState.set(driverId, current);
    return delta;
  }
}

// For frequent location updates:
const delta = broadcaster.computeDelta(driverId, newLocation);
// delta might be: { d: "d_12345", lt: 6.9271, ln: 80.7789, sp: 25 }
// Only 60 bytes instead of 140

On the client side, merge deltas into your local state:

let driverState: Record = {};

socket.on("driver_update", (delta) => {
  Object.assign(driverState, delta);
  // driverState now reflects the full state after each delta
});

Delta encoding works especially well for continuous data like GPS coordinates, sensor readings, or stock prices. I've seen 60–80% payload reduction in practice because most updates only touch 2–3 fields.

Compress with permessage-deflate carefully

WebSocket compression via the permessage-deflate extension can reduce payload size by 70–90% for repetitive data, but it has hidden costs. Compression consumes CPU on both server and client, and with high-frequency messages, the overhead can exceed the savings.

I enable compression selectively:

import WebSocket from "ws";

const wss = new WebSocket.Server({
  port: 8080,
  perMessageDeflate: {
    serverNoContextTakeover: true,
    clientNoContextTakeover: true,
    serverMaxWindowBits: 10,
    concurrencyLimit: 10,
    threshold: 1024 // Only compress messages larger than 1 KB
  }
});

The threshold setting is critical. Messages smaller than 1 KB rarely benefit from compression; the compression overhead eats gains. For a 100-byte message, compression might expand it to 120 bytes. Only compress bulk updates, batch operations, or infrequent large payloads.

In a trading system I built, I kept compression enabled for order history snapshots (which are sent once per day per client) but disabled it for real-time tick updates (which fire 50 times per second per symbol). The tick updates were already small after delta encoding, so compression was pure overhead.

Test compression settings under realistic load. I use serverMaxWindowBits: 10 (1 KB sliding window) as a safe default—it's easier on memory than the full 15-bit window, especially at high connection counts.

Measure bandwidth per connection

You can't optimize what you don't measure. I instrument every WebSocket connection to track incoming and outgoing bytes per second:

class ConnectionMonitor {
  private bytesOut = 0;
  private bytesIn = 0;
  private startTime = Date.now();

  recordOutgoing(bytes: number) {
    this.bytesOut += bytes;
  }

  recordIncoming(bytes: number) {
    this.bytesIn += bytes;
  }

  getBandwidth() {
    const elapsedSeconds = (Date.now() - this.startTime) / 1000;
    return {
      outKbps: (this.bytesOut * 8) / elapsedSeconds / 1000,
      inKbps: (this.bytesIn * 8) / elapsedSeconds / 1000,
      bytesOut: this.bytesOut,
      bytesIn: this.bytesIn
    };
  }
}

// Hook into ws library
wss.on("connection", (socket) => {
  const monitor = new ConnectionMonitor();

  socket.on("message", (data) => {
    monitor.recordIncoming(data.length);
  });

  // Intercept send to measure outgoing
  const originalSend = socket.send.bind(socket);
  socket.send = (data, callback) => {
    monitor.recordOutgoing(data.length);
    return originalSend(data, callback);
  };

  // Report every 60 seconds
  setInterval(() => {
    const bw = monitor.getBandwidth();
    console.log(`Connection ${socket.url}: ${bw.outKbps.toFixed(2)} Kbps out`);
  }, 60000);
});

Log this data to your monitoring system (Datadog, New Relic, CloudWatch). Track p50, p95, p99 bandwidth per connection. If you see p99 > 1 Mbps on a mobile client, you have a problem. If average is > 500 Kbps, consider optimization opportunities.

I also measure end-to-end: from Kafka producer to WebSocket subscriber. Add timestamps to messages in Kafka, then measure latency + payload size together:

// In Kafka producer
const message = {
  ts: Date.now(),
  d: driverId,
  lt: location.latitude,
  ln: location.longitude
};

// In WebSocket handler
socket.on("driver_update", (msg) => {
  const latencyMs = Date.now() - msg.ts;
  metrics.histogram("driver_update_latency", latencyMs);
  metrics.gauge("payload_size", JSON.stringify(msg).length);
});

Common mistakes

• Sending full objects every time. I've seen systems broadcast an entire user profile (50+ fields) on every login event. Deltas reduce this to 5 fields.
• Enabling compression globally without a threshold. Compressing tiny, high-frequency messages burns CPU with no benefit. Measure before enabling.
• Including debug or metadata fields in production. Accidentally shipping _debug_source, _internal_id, or stack_trace to clients inflates every payload. Use environment-based serialization.
• Forgetting to account for frame overhead. WebSocket frames add 2–14 bytes per message. If you send 1,000 1-byte updates instead of one 1,000-byte update, you're adding 14 KB of overhead. Batch when possible.
• Not monitoring per-connection bandwidth under load. Optimization in isolation doesn't matter. Test at 10K concurrent connections; behavior changes.

Key takeaways

• Every byte you send in a fan-out WebSocket system is replicated across thousands of connections. Small optimizations compound dramatically.
• Abbreviate JSON keys or switch to binary formats (MessagePack, Protocol Buffers) in high-throughput scenarios. The savings are 60–70% routinely.
• Implement delta encoding server-side for continuous data. Only broadcast fields that changed between updates.
• Use permessage-deflate selectively with a threshold. Don't compress small, frequent messages; the overhead exceeds savings.
• Instrument every connection to measure bandwidth. Track p95 and p99 per-connection outgoing bytes per second. Alert if they exceed your target.
• Test under realistic load (thousands of concurrent subscribers) before deploying optimizations to production.