Replica sets explained in MongoDB, NoSQL & Data Modeling

Replica sets explained in MongoDB, NoSQL & Data Modeling

When your MongoDB database goes down at 3 a.m., your team's already on call and your customers are seeing errors. That's where replica sets come in—they're the foundation of high availability in MongoDB. In this article, we'll walk through how replica sets work, from primary-secondary architecture to automatic failover and write durability, so you can build systems that stay online when things go wrong.

What is a replica set?

A MongoDB replica set is a group of mongod instances that maintain the same data set. One instance is the primary; the others are secondaries. The primary accepts all writes, applies them to its oplog (operation log), and secondaries pull from that oplog to replicate the data. If the primary fails, the replica set automatically elects a new primary from the secondaries, so your application keeps running.

Minimum viable setup is three nodes: one primary and two secondaries. The third node breaks ties during elections. You can run all three on separate machines, or use an arbiter (a lightweight member that doesn't store data) if budget is tight—though arbiters are generally reserved for smaller deployments.

Primary and secondaries — writes go to the primary; secondaries replicate

All writes in a replica set go to the primary. The primary writes the change to its local storage, records it in the oplog, and then secondaries asynchronously pull those oplog entries and apply them. This is the core contract: the primary is your source of truth for writes.

Let's say you're building an e-commerce platform and a customer places an order. That write hits the primary:

const db = client.db('shop');
const orders = db.collection('orders');

const result = await orders.insertOne({
  customerId: 'user_123',
  items: [{ sku: 'WIDGET_A', qty: 2, price: 29.99 }],
  total: 59.98,
  createdAt: new Date(),
  status: 'pending'
});

console.log('Order written:', result.insertedId);

That write lands on the primary. Meanwhile, the secondaries are continuously replicating from the oplog. If you have a secondary lagging behind—maybe it's doing a lot of I/O—it will eventually catch up. This asynchronous nature is why replica sets give you availability, not instant consistency across all members.

Reads from secondaries will see slightly stale data. In a fast, healthy cluster, that lag is milliseconds. In a stressed cluster, it could be seconds. You control this trade-off with read preference, which we'll cover later.

Automatic failover — a secondary is elected when the primary fails

The magic of replica sets is automatic failover. Each secondary periodically heartbeats the primary. If the primary doesn't respond for a configurable timeout (default 10 seconds), the secondaries hold an election. The secondary with the highest priority and most up-to-date oplog becomes the new primary.

Here's what that looks like in practice. You have three nodes:

At 14:32, mongo-primary crashes. The secondaries detect heartbeat failure within 10 seconds. They vote, and mongo-secondary-1 wins the election (it has the highest priority or was most up-to-date). It becomes primary. Your application reconnects and keeps writing.

During the election window (roughly 10–30 seconds), writes will fail because there's no primary. That's why you configure write concern to handle this gracefully—your app should retry writes or queue them locally.

The key config is in your replica set configuration:

// Example replica set init config (run on primary via mongosh)
// This sets priority to control election likelihood
rs.initiate({
  _id: 'myReplicaSet',
  members: [
    { _id: 0, host: 'mongo-primary:27017', priority: 3 },
    { _id: 1, host: 'mongo-secondary-1:27017', priority: 2 },
    { _id: 2, host: 'mongo-secondary-2:27017', priority: 1 }
  ],
  settings: {
    electionTimeoutMillis: 10000
  }
});

Higher priority nodes are preferred in elections. You can also set priority: 0 to prevent a node from ever becoming primary—useful for read-only secondaries in remote data centers.

Write concern — w: majority makes writes durable across members

Write concern tells MongoDB how many replica set members must acknowledge a write before the operation returns as successful. This is your durability guarantee.

The levels are:

• w: 1 (default) — Primary acknowledges; secondaries may not have replicated yet.
• w: majority — A majority of replica set members (including primary) have written to disk.
• w: 0 — Fire and forget; no acknowledgment (rarely used in production).

For the e-commerce order we inserted earlier, let's enforce w: majority to ensure the order is safe:

const orders = db.collection('orders');

const result = await orders.insertOne(
  {
    customerId: 'user_123',
    items: [{ sku: 'WIDGET_A', qty: 2, price: 29.99 }],
    total: 59.98,
    createdAt: new Date(),
    status: 'pending'
  },
  { writeConcern: { w: 'majority', wtimeout: 5000 } }
);

console.log('Order written and replicated to majority:', result.insertedId);

Now MongoDB won't confirm the write until the primary and at least one secondary have written it to disk. If a majority doesn't acknowledge within 5 seconds, the write throws an error and your app can retry.

The tradeoff: w: majority is slower than w: 1 because the primary must wait for secondaries. In a CRM storing customer interactions, that latency is usually acceptable (100–200ms extra). For a cache or session store, you might stick with w: 1.

You can set write concern per operation, per collection, or at the client level to avoid repeating yourself.

Read preference — read from secondaries for analytics, accepting some lag

By default, reads go to the primary. But you can route reads to secondaries if you accept stale data. This is powerful for offloading analytics, reporting, or heavy queries away from the primary.

Read preference modes are:

• primary (default) — Only the primary; consistent, but can overload it.
• primaryPreferred — Primary first, then secondaries; good for most app reads.
• secondary — Only secondaries; useful for analytics or read replicas.
• secondaryPreferred — Secondaries first, then primary; balances load and consistency.
• nearest — Lowest network latency; ignores primary/secondary role.

In your SaaS product, you might run analytics on a secondary to avoid hammering the primary:

const orders = db.collection('orders');

// Aggregation for daily revenue report—read from secondary
const dailyRevenue = await orders.aggregate([
  {
    $match: {
      createdAt: {
        $gte: new Date('2024-01-01'),
        $lt: new Date('2024-01-02')
      }
    }
  },
  {
    $group: {
      _id: null,
      totalRevenue: { $sum: '$total' },
      orderCount: { $sum: 1 }
    }
  }
], {
  readPreference: { mode: 'secondary' }
}).toArray();

console.log('Daily revenue (may be 1-2s old):', dailyRevenue);

The secondary might be a few seconds behind the primary. For a dashboard refreshed every minute, that's fine. For a customer viewing their invoice immediately after placing an order, you'd use primary read preference or wait for replication.

Common mistakes

• Ignoring write concern in critical paths. Relying on w: 1 for payment or order writes leaves you vulnerable to data loss if the primary crashes mid-replication. Use w: 'majority' for anything that can't be rewritten.
• Reading from secondaries without handling stale data. A secondary can lag during network issues or hardware stress. If your app can't tolerate old reads, stick with primary or implement polling/retry logic.
• Setting very short election timeouts. electionTimeoutMillis: 1000 causes constant elections in a flaky network. Stick with the default 10 seconds or increase it in unstable environments.
• Running only two data-bearing nodes. A two-node replica set loses high availability if either node fails. Use three nodes minimum, or add an arbiter.
• Not monitoring oplog size. The oplog is a capped collection. If a secondary falls too far behind and the oplog rolls over, it needs a full resync, which is slow and CPU-intensive. Monitor oplog lag and alert if it exceeds a threshold.

Key takeaways

• A replica set has a primary (accepts writes) and secondaries (replicate asynchronously). Elections automatically promote a secondary if the primary fails.
• Write concern w: 'majority' ensures a write is durable across a majority of members before returning. Use it for critical operations; accept the latency cost.
• Read preference lets you route reads to secondaries for analytics and reporting, reducing load on the primary. Accept that secondaries may lag by milliseconds to seconds.
• Run at least three nodes to avoid split-brain scenarios and ensure a quorum for elections. Monitor oplog lag and election frequency; they're early warning signs of cluster problems.
• Failover is automatic and typically takes 10–30 seconds. Your application must be resilient to temporary write failures during elections—implement retries or queue writes locally.