Where change data capture shines: lessons from production

Where change data capture shines

Database changes happen constantly in production systems—orders placed, user profiles updated, inventory counts adjusted. Instead of polling tables or writing custom triggers, you can turn those changes into streams that power real-time features across your entire platform. This is where change data capture (CDC) transforms your architecture from a collection of isolated subsystems into a cohesive, event-driven ecosystem.

I've built systems where CDC was bolted on as an afterthought, and I've designed platforms where it was baked in from day one. The difference is dramatic. CDC lets you decouple services, keep derived data fresh, and migrate legacy systems without rewriting everything at once. Over the past 5+ years working with .NET, NestJS, and event platforms, I've learned exactly where CDC creates the most value—and where it's overkill.

Why CDC matters: The problem it solves

Before diving into use cases, let's be clear about what CDC does. It captures every INSERT, UPDATE, and DELETE on your source database and publishes those changes as events. Tools like Debezium, AWS DMS, or custom solutions read your database transaction log (PostgreSQL's WAL, MySQL's binlog, SQL Server's transaction log) and emit events to Kafka, SQS, or your message broker of choice.

The core benefit: you remove the coupling between your application and downstream consumers. Your application doesn't know or care that Elasticsearch needs to stay in sync, or that your analytics warehouse is listening. Changes flow automatically, and any new consumer can catch up from the beginning.

Sync search indexes like Elasticsearch

Search is one of the clearest wins for CDC. Imagine you're running an e-commerce platform with millions of products. Every product update—price change, stock level, description—needs to hit Elasticsearch instantly so search results stay accurate.

Without CDC, you'd need to either:

• Make your application code responsible for syncing both PostgreSQL and Elasticsearch in the same transaction (coupled, brittle)
• Poll your database every few seconds looking for changes (wasteful, slow)
• Build a custom webhook system (reinventing the wheel)

With CDC, Debezium watches your PostgreSQL transaction log and streams product changes to Kafka. A consumer service reads those events and updates Elasticsearch. If Elasticsearch goes down, Kafka buffers the changes. When it comes back online, the consumer catches up automatically.

Here's a simple NestJS consumer that listens to product changes:

import { Injectable } from '@nestjs/common';
import { Kafka } from 'kafkajs';
import { ElasticsearchService } from '@nestjs/elasticsearch';

@Injectable()
export class ProductSyncService {
  private kafka: Kafka;
  private consumer;

  constructor(private elasticsearchService: ElasticsearchService) {
    this.kafka = new Kafka({
      clientId: 'product-sync-service',
      brokers: ['kafka:9092'],
    });
    this.consumer = this.kafka.consumer({ groupId: 'product-sync-group' });
  }

  async onModuleInit() {
    await this.consumer.connect();
    await this.consumer.subscribe({ topic: 'products.changes' });

    await this.consumer.run({
      eachMessage: async ({ topic, partition, message }) => {
        const change = JSON.parse(message.value.toString());

        if (change.op === 'c' || change.op === 'u') {
          // Create or update
          await this.elasticsearchService.index({
            index: 'products',
            id: change.after.id.toString(),
            body: {
              id: change.after.id,
              name: change.after.name,
              price: change.after.price,
              stock: change.after.stock,
              updatedAt: new Date(),
            },
          });
        } else if (change.op === 'd') {
          // Delete
          await this.elasticsearchService.delete({
            index: 'products',
            id: change.before.id.toString(),
          });
        }
      },
    });
  }
}

The operations are: c for create, u for update, d for delete. This pattern is dead simple and decoupled. Your application doesn't publish to Kafka—Debezium does it automatically. This means you can add new indexes or change Elasticsearch configuration without touching application code.

Build read models and caches

In event-driven systems, you often maintain read models—denormalized views of your data optimized for queries that matter to your users. A ride-hailing platform, for example, might maintain a read model of "drivers near location X with rating > 4.5." That's expensive to compute from normalized tables every request.

CDC feeds these read models continuously. When a driver's location updates in PostgreSQL, that change flows to Kafka. A consumer updates the read model in Redis or a document store. The application layer queries the read model, always hot and always consistent within a few milliseconds.

Consider a CRM system I built at my last company. The core data lived in normalized PostgreSQL tables. But the UI needed fast access to:

• Customer summary (name, email, total lifetime value, churn risk)
• Recent interactions (calls, emails, meetings)
• Next action items

Instead of joining five tables on every request, I used CDC to build a denormalized customer document in MongoDB whenever source tables changed. Read latency dropped from 200ms to 10ms.

The process looked like this:

1. Debezium captures changes from customers, interactions, and accounts tables
2. A NestJS consumer listens to all three topics
3. When any change arrives, it fetches the full customer context from PostgreSQL and writes a denormalized document to MongoDB
4. The API reads from MongoDB instead of PostgreSQL

This works beautifully because the consumer is idempotent. If the same event is processed twice (Kafka guarantees at-least-once delivery), MongoDB just overwrites the document with identical data. No corruption, no conflicts.

Feed analytics warehouses

Analytics teams need a complete, reliable history of data changes. "How many users signed up each day?" "Which products drove the most revenue?" These queries need access to raw events, not just current state.

CDC is perfect here because it's a natural fit for data warehouses. Every change becomes a row in your fact table or data lake. Tools like AWS Glue, Snowflake, or BigQuery can consume Kafka streams and insert them directly into warehouse tables.

At a travel booking platform I worked on, we needed to track every search, every click, every booking. Rather than have the application log events (which gets messy—you forget a field, you change a field's meaning, you lose events if the logger fails), we used CDC on the bookings, payments, and searches tables.

Kafka delivered changes to S3 via Kinesis Firehose. A Lambda job partitioned them by date and uploaded Parquet files. Athena queried the data lake directly. Our analytics team got a complete, authoritative history without any application-level event logging infrastructure.

This also meant that if we discovered a data quality issue (say, we weren't capturing the user's device type), we could backfill the data lake from the database transaction log. With custom event logging, that's often impossible.

Migrate legacy systems gradually

Legacy system migrations are painful. You can't flip a switch and move to a new architecture—you need a gradual, reversible process. CDC enables this.

I recently helped migrate a trading platform from a monolithic .NET application to microservices. The monolith used a single SQL Server database. We couldn't afford downtime, and we couldn't rewrite everything at once.

Our strategy:

1. Set up CDC on the monolith's database (SQL Server supports it natively)
2. Stream all changes to Kafka
3. Build new services (written in NestJS) that listen to Kafka and maintain their own PostgreSQL databases
4. Gradually migrate traffic from the monolith to the new services
5. Keep both systems in sync via CDC as a safety net

When the trading service was ready, we redirected writes to it, but the monolith's CDC feed kept its local cache warm. If something went wrong in the new service, we could instant-flip back to the monolith.

This wouldn't have been feasible without CDC. Custom point-to-point integration would have been fragile and hard to test. CDC gave us a clean, auditable migration path.

Common mistakes

Treating CDC events as the source of truth without proper idempotency. Kafka delivers messages at-least-once. If you're not idempotent in your consumer (e.g., you increment a counter instead of setting a state), you'll corrupt data on retries. Always design consumers to be idempotent.

Ignoring the initial snapshot. When you start CDC on a table with a million rows, you need to bootstrap consumers with those rows before streaming new changes. Tools like Debezium support snapshot modes—use them. Otherwise, your downstream systems will be missing historical data.

Not monitoring consumer lag. If your consumer falls behind (Kafka has 10,000 messages queued but the consumer is only processing 100/sec), you'll have stale data in your read model or search index. Monitor lag at all times.

Assuming CDC is real-time. It's very fast (typically milliseconds), but not instantaneous. If your application expects Elasticsearch to have the change immediately after writing to PostgreSQL, you might get stale results for a moment. Design UX accordingly.

Over-engineering the consumer. A simple NestJS or Go service that listens to Kafka and updates Redis or MongoDB is often enough. Don't build a framework. Keep it boring and maintainable.

Key takeaways

• CDC turns database transaction logs into event streams, decoupling your application from downstream consumers
• Use CDC to keep search indexes (Elasticsearch) in sync without application-level coupling
• Build denormalized read models and caches from CDC events to optimize query latency
• Stream CDC events to data warehouses for reliable, complete analytics history
• Leverage CDC during system migrations to run old and new systems in parallel safely
• Always make consumers idempotent—Kafka delivers messages at-least-once, not exactly-once
• Monitor consumer lag obsessively; stale read models are worse than slow ones
• CDC is not real-time (milliseconds, yes; instant, no)—design your UI to tolerate that latency