Read replicas: the fine print — best practices explained

Read replicas: the fine print — best practices explained

Read replicas promise cheap scaling: point your read queries at standby databases and watch your throughput multiply. But the moment you deploy them to production, you discover the fine print. Replication lag—that tiny delay between writes hitting the primary and reaching replicas—creates a class of bugs that only appear under load and only affect your most engaged users.

We've seen this pattern across e-commerce checkouts, trading platforms, and SaaS dashboards. The architecture feels elegant until a user edits their profile, refreshes the page, and sees stale data. Today we'll walk through what read replicas actually are, where they shine, and the one gotcha that bites everyone.

Writes go to the primary; reads spread across replicas

The fundamental principle is straightforward. Your primary database accepts all writes—inserts, updates, deletes. It's the single source of truth. Read replicas are followers that stream the binary log from the primary and apply the same changes asynchronously. Reads fan out across all replicas, distributing load.

On paper, this halves database CPU for a read-heavy workload. In practice, if you're doing 70% reads and 30% writes, you add a replica and suddenly your read capacity doubles while write capacity stays the same. Add another replica and you've tripled read capacity for the cost of roughly 1.5 times the total compute.

The architecture looks like this:

• User submits a form → write to primary
• Analytics query runs → read from replica 1
• Dashboard loads user's data → read from replica 2
• Background job fetches batch of records → read from replica 3

The primary handles all mutations. Replicas handle all reads. Simple. But there's a catch that changes everything.

Replication lag means your own write may not be visible yet

Replication lag is the delay between when a write completes on the primary and when it appears on replicas. It's usually milliseconds, but under load it can stretch to seconds. More importantly: you can't control it. You can monitor it, but you can't guarantee it's zero.

This creates a specific failure mode. A user submits an order. The API writes to the primary and returns 200 OK. The user's browser refreshes and queries a replica for the latest orders. The replica hasn't caught up yet. The user sees an empty order list and panics—they think the order failed.

We saw this in a travel booking system. Users would book a flight, get a confirmation page, navigate away, come back, and the booking wouldn't appear in their history for 2-3 seconds. Support tickets flooded in. The issue wasn't the database—it was replication lag.

The timing matters:

1. User's browser sends POST to /orders with flight details
2. API receives request, validates, writes to primary database (1ms)
3. API returns 201 Created with order ID (2ms total)
4. User's browser immediately GETs /orders/history from replica (3ms total)
5. Replica is still applying the write (arrives at 4ms)
6. User sees empty list

The query succeeded. The data was written. But the user can't see their own write because the replica lagged. This isn't a bug in your code—it's a guarantee you need to encode into your architecture.

Send read-your-own-writes traffic to the primary

The solution is to route user-facing reads that follow a user's own writes back to the primary. This is called read-your-own-writes consistency (or sometimes causal consistency). You don't need to send all reads to the primary—just the ones that touch data the user just modified.

The pattern is straightforward. After a write succeeds, store a flag in the user's session or local storage indicating which tables were modified. For the next 5-10 seconds, any read query touching those tables routes to the primary. Once the read succeeds, you know replication has caught up and future reads can hit replicas again.

Here's a practical example in a NestJS API:

// Middleware to track user writes
app.use((req, res, next) => {
  if (!req.session.dirtyTables) {
    req.session.dirtyTables = new Map();
  }
  next();
});

// After a user updates their profile
async updateProfile(userId: string, updates: any) {
  const result = await this.db.primary
    .update('users')
    .set(updates)
    .where({ id: userId })
    .execute();
  
  // Mark this table as dirty for this user
  this.req.session.dirtyTables.set('users', Date.now() + 10000);
  return result;
}

// For reads, check if we need primary
async getProfile(userId: string) {
  const isDirty = this.req.session.dirtyTables.has('users');
  const expiredDirty = isDirty && 
    this.req.session.dirtyTables.get('users') < Date.now();
  
  const connection = (isDirty && !expiredDirty) 
    ? this.db.primary 
    : this.db.replica;
  
  const user = await connection
    .select('*')
    .from('users')
    .where({ id: userId })
    .first();
  
  // If we hit primary and got data, replica should have it soon
  if (isDirty && !expiredDirty) {
    this.req.session.dirtyTables.delete('users');
  }
  
  return user;
}

You could also use a version vector or timestamp-based approach. After a write, the primary returns a version number. Client stores it and passes it on subsequent reads. If the replica's version is ahead of the stored version, the replica is fresh enough.

The key principle: don't guess where replication lag is safe to ignore. Make it explicit in your code.

Dashboards and reports are perfect for replicas

The best use case for read replicas is exactly the opposite scenario: queries where the user doesn't care about freshness and replication lag is irrelevant.

Dashboards that show weekly metrics, monthly revenue, user cohorts—these are ideal replica workloads. A 30-second delay in a weekly dashboard doesn't matter. A 2-hour delay in a batch analytics job doesn't matter. These queries are often expensive (full table scans, complex joins, aggregations) and you especially want them off the primary.

Background jobs are another perfect fit. A job that processes customer emails, expires sessions, calculates recommendations—it can tolerate stale data. If it reads from an hour-old replica snapshot, that's usually fine.

In a CRM system we worked on, the primary was under pressure from transactional reads (users loading contact details, updating fields). We moved the reporting dashboard to a dedicated read replica and the primary's CPU dropped 30%. The dashboard showed data that was 5-10 seconds stale. Nobody noticed or cared.

The distinction:

• Needs primary: user editing their own data, then immediately reading it back; checkout flows; critical state checks
• Needs replica: analytics dashboards; batch processing; user browsing data they didn't just change; reports and exports

Common mistakes

Mistake 1: Assuming replication lag is always zero. It's not. Even a modern RDS replica in the same AZ can lag under load. Build for 100ms lag, not 1ms.

Mistake 2: Spreading all reads across replicas without thinking. You need a deliberate routing strategy. The default should be: reads go to replica unless this specific query needs to see fresh data.

Mistake 3: Not monitoring replication lag. Set CloudWatch alarms on replica lag. When lag spikes (often before you see customer complaints), you'll know why. Without visibility, you'll debug the wrong thing.

Mistake 4: Using replicas for cache eviction checks. If your code checks a replica to see if a cache key exists before updating it, you'll get false positives and stale cache. Always check the primary for cache invalidation logic.

Mistake 5: Promoting a replica without understanding its lag.** If the primary fails and you promote a lagged replica, you lose recent writes. This isn't a read replica problem—it's a failover strategy problem. Know your RPO (recovery point objective).

Key takeaways

• Read replicas are asynchronous followers of your primary. Writes go to primary, reads fan out to replicas. This scales reads cheaply.
• Replication lag means users can write data and not see it immediately when reading from a replica. This surprises everyone in production.
• Route reads that follow a user's own writes back to the primary. Use session-based dirty flags or version vectors to keep this simple and automatic.
• Dashboards, reports, and batch jobs are ideal replica workloads. They don't need fresh data and they're often the most expensive queries.
• Monitor replication lag constantly. Set alarms. Understand your tolerance. Don't assume zero lag.
• Read replicas are a multiplier on your read capacity, but only if you route traffic deliberately. The architecture is simple; the operational discipline is not.