Pick a compression codec: a hands-on guide for engineers

Pick a compression codec: a hands-on guide for engineers

Most teams leave throughput on the table by not compressing Kafka messages. Compression isn't a new trick—it's a proven way to squeeze 3–10x more data through your brokers without adding hardware. In this guide, we'll walk through the compression codecs available to you, show you when each one wins, and help you integrate them into your producer configuration today.

Why compression matters for your throughput

Kafka measures throughput in bytes per second. When you compress messages before sending them, you're reducing network I/O, disk I/O, and storage footprint in one move. A 10 MB batch of log lines might compress to 1–2 MB. That's the same semantic data flowing through your cluster at a fraction of the cost.

The tradeoff is CPU: your producers burn cycles compressing, your brokers store compressed payloads, and your consumers burn cycles decompressing. On modern hardware with multi-core processors, that CPU cost is almost always worth it. Network and disk I/O are your true bottlenecks in most Kafka setups, not CPU.

Three codecs ship with Kafka by default: gzip, lz4, and zstd. Snappy and other options exist but rarely show up in production. We'll focus on the three that matter.

zstd: best ratio, good speed

Zstandard (zstd) is the newest addition to Kafka's compression arsenal and represents the evolution of compression thinking over the past decade. It was designed from the ground up to hit a sweet spot: compression ratio rivaling gzip, decompression speed often faster than lz4, and tunable CPU cost.

In real-world testing across trading systems, e-commerce order events, and CRM updates, zstd typically compresses to 15–25% of the original size for structured text data like JSON. For binary or already-compressed payloads (images, video), you'll see less benefit, but the overhead is minimal.

Here's how to enable zstd in a NestJS producer using the KafkaJS library:

import { Kafka, CompressionTypes } from 'kafkajs';

const kafka = new Kafka({
  clientId: 'my-app',
  brokers: ['localhost:9092'],
});

const producer = kafka.producer();

await producer.connect();

await producer.send({
  topic: 'orders',
  compression: CompressionTypes.ZSTD,
  messages: [
    {
      key: 'order-123',
      value: JSON.stringify({
        orderId: '123',
        customerId: 'cust-456',
        items: [...],
        total: 299.99,
      }),
    },
  ],
});

The CPU cost of zstd is tunable via broker-side configuration. By default, Kafka uses a moderate compression level. For write-heavy topics where you can afford a bit more CPU at the producer, you can push the compression level higher and achieve 20–30% ratio improvements. For latency-sensitive producers, back it off.

One caveat: zstd requires Kafka 2.1.0 or later on all brokers and clients. If your cluster is still on 1.1.x, you're stuck with gzip or lz4. Most teams running modern Kafka (2.6+, 3.x) have no barrier to adoption.

lz4: very fast, good default

LZ4 is the speed champion. It trades some compression ratio for decompression that runs at gigabytes per second on modern CPUs. If your producers are CPU-constrained or your network is already fast enough, lz4 is the sensible default.

In production ride-hailing and travel systems, lz4 consistently compresses JSON event streams to 30–40% of original size while adding almost zero producer latency. Decompression is so fast that even low-power edge devices or containers with CPU limits rarely notice it.

Enable lz4 in your producer:

await producer.send({
  topic: 'events',
  compression: CompressionTypes.LZ4,
  messages: [
    {
      key: 'event-' + Date.now(),
      value: JSON.stringify({
        userId: 'user-789',
        action: 'purchase',
        timestamp: new Date().toISOString(),
      }),
    },
  ],
});

LZ4 is battle-tested across every major cloud platform and has been Kafka's default for years. It's simple, predictable, and fast. If you're uncertain which codec to pick and you have no specific throughput crisis, lz4 is your answer.

The downside: compression ratio lags behind zstd by 20–40% on typical workloads. For very high-throughput clusters where every megabyte counts, that gap adds up.

Compression works on batches—batch well

Compression in Kafka is not message-by-message; it operates on batches. A batch is a collection of messages the producer bundles together before sending. Compression algorithms thrive on repetition and context. A single 100-byte JSON object won't compress much. But 1,000 similar objects in a batch compress magnificently because the algorithm finds patterns across all of them.

To maximize compression gains, tune your producer batching settings. In KafkaJS, this means controlling the linger time and batch size:

const producer = kafka.producer({
  idempotent: true,
  maxInFlightRequests: 5,
  compression: CompressionTypes.ZSTD,
  transactionTimeout: 30000,
});

await producer.send({
  topic: 'events',
  compression: CompressionTypes.ZSTD,
  messages: events.map((e) => ({
    key: e.userId,
    value: JSON.stringify(e),
  })),
  timeout: 30000,
});

The producer will accumulate messages until either the batch reaches a size threshold (default 16 KB) or the linger time expires (default 0 ms, meaning send immediately). If linger is 0, most batches are small, and compression wastes effort on tiny payloads.

Increase linger to 10–50 ms in scenarios where you tolerate a bit of latency for better batching. That small delay lets the producer collect 10–100x more messages per batch, and compression kicks in with full force. On high-throughput topics, this alone can cut network I/O by half.

Here's a practical example for an SaaS metrics pipeline:

const producer = kafka.producer({
  compression: CompressionTypes.ZSTD,
  allowAutoTopicCreation: false,
});

const lingerMs = 20; // wait up to 20 ms to batch more messages
let buffer = [];

setInterval(async () => {
  if (buffer.length === 0) return;

  await producer.send({
    topic: 'metrics',
    messages: buffer,
    timeout: 5000,
  });

  buffer = [];
}, lingerMs);

// Collect events into buffer
app.post('/track', (req, res) => {
  buffer.push({
    key: req.body.userId,
    value: JSON.stringify({
      event: req.body.event,
      timestamp: Date.now(),
      properties: req.body.properties,
    }),
  });

  res.json({ ok: true });
});

With this approach, you're sending 100–500 events per batch instead of 1–2, and compression transforms a 100 KB payload into 10–15 KB. Over a day, that's terabytes saved on storage and network.

Consumers decompress automatically

The beautiful part of Kafka compression: your consumers don't need to know or care. When a consumer subscribes to a topic, it receives compressed batches from the broker. The Kafka client library automatically detects the compression codec (it's stored in the batch metadata) and decompresses transparently.

In other words, you enable compression on the producer side and forget about it on the consumer side. Your consumer code remains unchanged:

const consumer = kafka.consumer({
  groupId: 'order-processor',
  sessionTimeout: 20000,
});

await consumer.connect();
await consumer.subscribe({ topic: 'orders' });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    // message.value is automatically decompressed
    const order = JSON.parse(message.value.toString());
    console.log('Processing order:', order.orderId);
    // no compression code needed here
  },
});

This design removes friction from your team. A consumer team doesn't need to update their code when a producer team switches from lz4 to zstd. The broker handles the translation. You can have heterogeneous compression across different topics without coordination.

That said, decompression does consume CPU on the consumer side. In latency-critical systems—real-time dashboards, trading engines—monitor whether decompression becomes a bottleneck. Usually it doesn't. On a 16-core consumer instance processing millions of messages per second, decompression overhead is a few percent. But in a severely CPU-constrained edge environment or a consumer pulling millions of tiny messages, it's worth profiling.

Common mistakes

Mistake 1: Compressing tiny batches. Setting compression codec but leaving linger at 0 means most batches are single or double-digit message counts. Compression adds overhead with minimal benefit. Fix: increase linger to 10–50 ms or batch messages in your application before sending.

Mistake 2: Forgetting the broker-side setting. Some teams set compression on the producer but the broker's log.compression.type is still set to producer (the default). This is actually correct—the broker respects the producer's codec choice. However, if the broker's setting is none, the broker will decompress and re-compress on re-replication, wasting CPU. Verify your broker config matches your producer strategy.

Mistake 3: Choosing codec based on benchmark alone. Zstd looks best on paper, but if your producer is already CPU-bound, switching from lz4 to zstd may increase latency. Measure end-to-end latency and throughput in a production-like test. The "best" codec is the one that meets your latency SLO while maximizing compression.

Mistake 4: Not monitoring decompression latency. Some teams compress with zstd and later discover that their consumer lag is growing because decompression is slow. (This usually happens when the compression level is too aggressive.) Monitor consumer CPU and per-message decompression time.

Key takeaways

• Compression is nearly free throughput. It trades a small amount of CPU for large reductions in network I/O and disk footprint. Most clusters are I/O bound, not CPU bound.
• zstd is the best choice for new systems: superior compression ratio, fast decompression, and tunable CPU cost. Requires Kafka 2.1+.
• lz4 is the safe default: extremely fast, lower CPU cost, battle-tested across industries. Use it if you're uncertain or if CPU is genuinely constrained.
• Compression operates on batches, so batch well. Increase linger time to 10–50 ms and let the producer accumulate messages. This amplifies compression gains tenfold.
• Consumers decompress automatically. Enable compression once on the producer side and all consumers benefit without code changes.
• Measure your own workload. JSON compresses better than binary. Homogeneous events compress better than heterogeneous ones. Test codec choice on realistic data.