Log compaction: a Kafka & WebSockets guide for developers

Opening: Why log compaction matters

In a trading platform, you're updating user balances hundreds of times per second. In an e-commerce system, product prices change constantly. In a SaaS application, customer configurations shift with every feature toggle. If you're using Kafka to stream these updates, you probably don't want to keep every single historical change—you want the current state. That's where log compaction: a Kafka & WebSockets guide for developers comes in. Log compaction is a cleanup policy that transforms Kafka topics into something closer to a table, retaining only the latest value for each key and discarding the older ones.

What log compaction does: the mechanics

By default, Kafka topics are retention-based: messages expire after a certain age or when the log reaches a size limit. Log compaction flips this model. Instead of time-based cleanup, Kafka looks at message keys and keeps only the most recent value per key, regardless of age. Older messages with the same key are removed during the compaction process.

Here's how it works in practice:

• A producer sends { key: "user:123", value: { balance: 100 } } at time T1
• At time T2, the same key is updated: { key: "user:123", value: { balance: 150 } }
• At time T3, another update: { key: "user:123", value: { balance: 200 } }
• Log compaction runs, removing the T1 and T2 messages, leaving only the T3 message

The compaction process doesn't happen instantly. Kafka's compaction runs in the background on a configurable schedule. You configure a topic with cleanup.policy=compact, and Kafka's log cleaner thread periodically scans the log to remove older entries.

To enable log compaction on a topic, use the broker configuration:

// Creating a compacted topic with the Admin client
const admin = kafka.admin();
await admin.createTopics({
  topics: [
    {
      topic: 'user-profiles',
      numPartitions: 3,
      replicationFactor: 2,
      configEntries: [
        { name: 'cleanup.policy', value: 'compact' },
        { name: 'min.cleanable.dirty.ratio', value: '0.5' },
        { name: 'segment.ms', value: '60000' }
      ]
    }
  ]
});

Key configuration parameters:

• cleanup.policy=compact: Enables log compaction for the topic
• min.cleanable.dirty.ratio: Triggers compaction when 50% of the log contains outdated messages (default is 0.5)
• segment.ms: How often Kafka closes a log segment (default 7 days). Compaction only runs on closed segments
• min.compaction.lag.ms: Minimum age of messages before they're eligible for compaction (default 0)

Real-world use cases for compacted topics

Log compaction shines when you're tracking mutable state that changes frequently but where only the current value matters.

User profiles and account data

In a ride-hailing platform, each driver's profile—rating, vehicle info, availability—changes throughout the day. Instead of keeping every update, you compact the topic and retain only the latest profile per driver ID. New onboarding services can read from the beginning of the compacted topic to bootstrap their view of all drivers instantly.

Product prices and inventory

An e-commerce catalog publishes price updates to a compacted topic keyed by SKU. Every time a price changes, you send a new message. Compaction keeps only the current price per product. Mobile apps and recommendation engines can consume this topic to stay synchronized with live pricing without wading through historical data.

Configuration and feature flags

A SaaS platform might stream configuration changes (feature toggles, rate limits, feature entitlements) to all services via a compacted topic. Each config key—say feature:new_checkout_flow—has only its current value retained. Every new service instance that starts up can replay the entire topic and reconstruct the current configuration state in milliseconds.

Entity state snapshots

In a CRM, contact records, deal stages, and account data change frequently. A compacted topic with key contact:12345 holds the latest contact details. When a new analytics or reporting service comes online, it reads the compacted topic and loads the current state of all entities without replaying years of history.

Tombstones: how deletion works in compacted topics

Log compaction solves the update problem, but what about deletes? In a regular log, you'd just send a new message. In a compacted topic, you need a special value to indicate "delete this key."

A tombstone is a message with a key but a null value. When Kafka's log cleaner encounters a tombstone, it marks that key for deletion. After compaction, both the tombstone and all older messages for that key are removed entirely.

Here's a practical example with a user lifecycle:

const producer = kafka.producer();
await producer.connect();

// User created
await producer.send({
  topic: 'user-profiles',
  messages: [
    {
      key: 'user:42',
      value: JSON.stringify({
        id: 42,
        name: 'Alice',
        email: 'alice@example.com',
        created: new Date().toISOString()
      })
    }
  ]
});

// User updated
await producer.send({
  topic: 'user-profiles',
  messages: [
    {
      key: 'user:42',
      value: JSON.stringify({
        id: 42,
        name: 'Alice Brown',
        email: 'alice.brown@example.com',
        created: new Date().toISOString()
      })
    }
  ]
});

// User deleted (tombstone)
await producer.send({
  topic: 'user-profiles',
  messages: [
    {
      key: 'user:42',
      value: null  // null value = tombstone
    }
  ]
});

On the consumer side, handle tombstones by removing the key from your in-memory state:

const consumer = kafka.consumer({ groupId: 'profile-cache-builder' });
await consumer.subscribe({ topic: 'user-profiles' });

const profileCache = new Map(); // in-memory state store

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const key = message.key.toString();
    
    if (message.value === null) {
      // Tombstone: delete this key
      profileCache.delete(key);
      console.log(`Deleted profile: ${key}`);
    } else {
      // Update: store the latest value
      const profile = JSON.parse(message.value.toString());
      profileCache.set(key, profile);
      console.log(`Updated profile: ${key}`, profile);
    }
  }
});

Tombstones are also important for compliance. If a user requests deletion under GDPR, sending a tombstone ensures their data is completely removed from the compacted topic after the next compaction run.

Rebuilding state: consumers and compacted topics

One of the most powerful features of log compaction is that new consumers can quickly rebuild their entire state by reading a compacted topic from the start. Because compaction keeps only the latest value per key, a fresh consumer gets a complete snapshot of the current system state without having to replay months or years of historical updates.

Here's a realistic scenario: you're deploying a new cache service that needs to know the current configuration and inventory levels across your entire system. Rather than querying a database, you create a consumer group that reads the compacted topics from offset 0 (the beginning).

const consumer = kafka.consumer({ 
  groupId: 'config-cache-init',
  sessionTimeout: 30000
});

const configCache = new Map();
let isReady = false;

await consumer.subscribe({ topic: 'system-config' });

await consumer.run({
  eachMessage: async ({ message }) => {
    const key = message.key.toString();
    if (message.value === null) {
      configCache.delete(key);
    } else {
      const config = JSON.parse(message.value.toString());
      configCache.set(key, config);
    }
  },
  // Detect when we've reached the end of the compacted topic
  eachPartition: async (partition) => {
    const highWaterMark = await consumer.getHighWater(partition.topic, partition.partition);
    const currentOffset = partition.offset;
    if (currentOffset >= highWaterMark - 1) {
      isReady = true;
      console.log('Cache rebuilt from compacted topic');
    }
  }
});

This approach is faster and simpler than building state from a database join or a REST API call. The compacted topic becomes a system-wide source of truth for current state.

Common mistakes to avoid

Not setting a key: Compaction only works on keyed messages. If your producer sends messages without a key (or with random keys), compaction can't deduplicate them. Every update becomes a new message. Always design your key strategy upfront—often a business identifier like user:123 or product:SKU.

Forgetting about tombstones in clients: A consumer that doesn't handle null values will crash or store broken state. Always check for null/null values when processing compacted topics. This is especially critical in WebSocket servers that broadcast state updates to clients—sending a null value to the front end might break the UI.

Waiting for compaction to happen immediately: Log compaction is asynchronous and scheduled. It doesn't run every second. If you're testing, understand that older messages might still exist in your topic for a while. Use min.compaction.lag.ms and min.cleanable.dirty.ratio to tune compaction frequency for your use case, but don't expect real-time cleanup.

Misunderstanding compaction and retention: Compaction and retention are different policies. You can mix them: cleanup.policy=compact,delete means "compact by key, but also delete messages older than X days." Without a retention policy on a compacted topic, old messages might never be deleted—only deduplicated.

Ignoring partition order: Compaction happens per partition. If you have multiple partitions and don't use consistent keying, the same logical entity might appear in different partitions, and compaction won't deduplicate across partitions. Design your key so messages for the same entity always go to the same partition (using Kafka's key-based partitioning).

Key takeaways

• Log compaction keeps only the latest message per key, making Kafka topics act like mutable tables instead of append-only logs
• Enable it with cleanup.policy=compact and tune when it runs with min.cleanable.dirty.ratio and segment.ms
• Ideal for mutable state: user profiles, prices, configuration, entity snapshots, and feature flags
• Use tombstones (null values) to delete keys; handle them explicitly on the consumer side
• New consumers can rebuild full system state by reading a compacted topic from the beginning, avoiding database queries
• Compaction is asynchronous—expect a delay between sending updates and them being compacted away
• Always use meaningful, consistent keys so compaction can deduplicate effectively
• Mix compaction and retention policies when compliance or data lifecycle requirements demand it