Compacted topics as a source of truth: a practical guide

Compacted topics as a source of truth

When you're running distributed systems at scale, keeping state synchronized across services is one of your hardest problems. Kafka's log compaction feature—enabled via cleanup.policy=compact—turns a topic into a persistent, replayable source of truth where every key always holds its latest value. In this post, we'll explore how to architect around compacted topics, when to use them, and how to combine them with time-based retention for production resilience.

Understanding cleanup.policy=compact

By default, Kafka deletes old messages based on time or size limits. With log compaction enabled, Kafka instead keeps the most recent value for each key, discarding older versions. This creates a fundamentally different guarantees model.

Log compaction runs asynchronously in the background. It doesn't delete messages immediately—instead, it marks segments as candidates for compaction, removing only the earlier values while preserving the latest. The offset sequence is preserved, so when a consumer replays the topic, it sees the final state of every key that was ever written.

This is powerful because:

• Any new consumer can replay the entire topic and reconstruct current state without needing a separate database.
• You have a complete audit trail in your event log, not just the current snapshot.
• Services can rebuild their local cache at any time by consuming from the beginning.

Consider an e-commerce platform where you track user profiles. Instead of storing user data in a database and syncing it to a cache, you could emit profile updates to a compacted topic. Each key is the user ID; each value is the full profile object. Old profile versions are compacted away, but the topic remains a complete source of truth for current profiles.

// Producer: emit user profile updates
import { Kafka } from 'kafkajs';

const kafka = new Kafka({
  clientId: 'user-service',
  brokers: ['localhost:9092'],
});

const producer = kafka.producer();
await producer.connect();

// Emit a user profile update with user ID as key
await producer.send({
  topic: 'user-profiles',
  messages: [
    {
      key: 'user-12345',
      value: JSON.stringify({
        userId: 'user-12345',
        name: 'Alice Johnson',
        email: 'alice@example.com',
        tier: 'premium',
        updatedAt: new Date().toISOString(),
      }),
    },
  ],
});

The topic configuration would look like:

// Create compacted topic
// Using Kafka Admin API
const admin = kafka.admin();
await admin.connect();

await admin.createTopics({
  topics: [
    {
      topic: 'user-profiles',
      numPartitions: 3,
      replicationFactor: 3,
      configEntries: [
        { name: 'cleanup.policy', value: 'compact' },
        { name: 'min.cleanable.dirty.ratio', value: '0.5' },
        { name: 'delete.retention.ms', value: '86400000' }, // 24 hours
      ],
    },
  ],
});

Every key keeps its newest value

The core mechanic of log compaction is simple but profound: for any given key, Kafka retains only the most recent message. Older versions with the same key are eligible for removal during compaction cycles.

This works well for reference data that changes but where you only care about the current state. User settings, feature flags, product catalogs, or payment method records are all good candidates.

However, there's a subtle point: compaction is not immediate. You configure how aggressively Kafka compacts via settings like min.cleanable.dirty.ratio (the proportion of the log that must be "dirty"—i.e., have a newer version—before compaction kicks in) and log.retention.bytes. This means for a short window, you might see duplicate keys in the log. Consumers reading from the beginning will still see all versions until compaction runs.

In practice, this rarely matters. By the time a new consumer starts reading, compaction has likely already run on the earlier segments. But in tests or during rapid topic replay, you might see stale values before the compacted version. Always code defensively by taking the last value for each key when rebuilding state:

// Consumer: rebuild state from compacted topic
import { Kafka } from 'kafkajs';

const kafka = new Kafka({
  clientId: 'cache-builder',
  brokers: ['localhost:9092'],
});

const consumer = kafka.consumer({ groupId: 'cache-rebuild-group' });
await consumer.connect();

const userCache = new Map();

await consumer.subscribe({ topic: 'user-profiles', fromBeginning: true });

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const userId = message.key.toString();
    const profile = JSON.parse(message.value.toString());
    
    // Always overwrite with the latest; compaction ensures minimal duplicates
    userCache.set(userId, profile);
  },
});

// After consuming from beginning, userCache holds the latest state
console.log('Cache rebuilt from compacted topic');

One important consideration: if a key is deleted (tombstone message with null value), log compaction will eventually remove all records for that key. If you want to preserve the fact that a key was deleted, you need to handle tombstones explicitly or use a different cleanup policy.

Great for caches and lookups

Compacted topics shine when you need a distributed cache that survives restarts and doesn't require a separate cache service. Multiple consumers can independently rebuild the same cache from the topic, ensuring consistency.

Consider a microservices architecture with a product catalog. Instead of every service querying a shared database, you could emit product updates to a compacted topic. Each service consumes from the beginning, builds its local cache, and then subscribes to new updates. If a service crashes, it replays from its last committed offset and catches up instantly.

A real-world scenario: a ride-hailing platform needs driver location and status across multiple services. The driver service publishes driver state changes to a compacted driver-state topic (key: driver ID, value: location, status, vehicle info). The matching service, notification service, and analytics service all consume from the same topic. Each maintains a local cache of driver state. When a driver updates their status, all services see it via Kafka without direct service-to-service calls.

The pattern is: produce once (correctly), consume many (cheaply). This inverts the typical database query pattern and scales beautifully.

Combine with retention for compact plus delete

You can set cleanup.policy=compact,delete to apply both compaction and time-based retention. This is useful when you want the latest state per key, but you also want to discard data older than a certain age for compliance or storage reasons.

For example, a SaaS platform might need to retain the latest billing information for each customer (compaction), but must delete all data for customers who have churned for over a year (time-based retention). With both policies, you get:

• Compaction: keeps the newest billing record per customer key.
• Deletion: removes all records older than 365 days.

The order matters. Kafka applies deletion first (based on retention.ms or retention.bytes), then compaction. So if you set retention.ms=86400000 (24 hours) and cleanup.policy=compact,delete, any message older than 24 hours is eligible for deletion regardless of whether it's the latest version for its key. Only messages within the retention window are compacted.

// Topic with both compaction and time-based deletion
await admin.createTopics({
  topics: [
    {
      topic: 'billing-records',
      numPartitions: 3,
      replicationFactor: 3,
      configEntries: [
        { name: 'cleanup.policy', value: 'compact,delete' },
        { name: 'retention.ms', value: '31536000000' }, // 365 days
        { name: 'min.cleanable.dirty.ratio', value: '0.5' },
      ],
    },
  ],
});

This hybrid approach is also useful for audit compliance. You keep a complete log for the retention period (satisfying audit requirements), but compaction prevents the log from growing infinitely for long-lived keys. A customer's latest order can be compacted to a single record, but the entire log is still discarded after 7 years if that's your policy.

Common mistakes

Assuming compaction is immediate. Compaction runs on a schedule based on the cleaner thread configuration. Don't assume old values disappear instantly. If you're testing or need to verify state, add explicit waits or check compaction metrics.

Using compacted topics for high-volume transactional data. Compacted topics are excellent for reference data but poor for event streams where you need every event. A trade execution log should not be compacted—you need every trade, not just the latest per trader. Use standard cleanup.policy=delete for those.

Ignoring tombstones. A message with a null value is a tombstone, signaling deletion. If you deserialize blindly, you'll crash. Always check for null values and handle them as deletes:

await consumer.run({
  eachMessage: async ({ topic, partition, message }) => {
    const key = message.key.toString();
    
    if (message.value === null) {
      // Tombstone: this key is deleted
      userCache.delete(key);
    } else {
      const profile = JSON.parse(message.value.toString());
      userCache.set(key, profile);
    }
  },
});

Not versioning value schemas. As your application evolves, your data schema will change. Compact topics store the latest version only, so a consumer replaying from the beginning will see mixed schema versions. Use versioning (e.g., Avro, Protobuf) or include a version field in your JSON.

Forgetting that compaction doesn't guarantee immediate deletion. Disk space might not be freed immediately after compaction. The segment is marked for deletion, but the actual space reclamation is asynchronous and depends on the broker's log cleanup schedule.

Key takeaways

• Compacted topics with cleanup.policy=compact retain only the latest value per key, making them ideal sources of truth for reference data.
• Any consumer can replay a compacted topic from the beginning to rebuild complete current state without a separate database.
• Use compacted topics for caches, configuration, and reference data (user profiles, feature flags, product info)—not for transactional event streams.
• Compaction is asynchronous; code defensively by always taking the last value for each key when rebuilding state.
• Combine cleanup.policy=compact,delete for hybrid retention: keep the latest state but discard old data beyond a time threshold.
• Handle tombstones (null values) explicitly; they signal key deletion after compaction.
• Version your schemas to handle evolution safely; compacted topics only store the latest version.
• Monitor compaction metrics and tune min.cleanable.dirty.ratio to control how aggressively the cleaner runs.