Retryable writes and reads: patterns, pitfalls and fixes

Retryable writes and reads: patterns, pitfalls and fixes

Network hiccups happen. A temporary connection drop, a server restart, or a brief DNS resolution failure can interrupt your MongoDB operations at the worst possible moment. Rather than forcing you to implement retry logic from scratch, modern MongoDB drivers ship with automatic retry mechanisms built in—but only if you understand how they work and what constraints they impose.

Why retryable operations matter in production

We've all been there: a database operation fails, logs show a connection timeout, and your team scrambles to debug whether it's a real problem or just a blip. Retryable writes and reads shift this burden away from manual retry code scattered across your codebase. When a transient network error occurs during a supported operation, the driver automatically retries once without your application knowing about it.

This is especially critical in microservices architectures where network partitions are inevitable. Instead of implementing circuit breakers and retry handlers for every single database call, you get sensible defaults out of the box. But sensible defaults aren't a silver bullet—they require you to understand their boundaries.

On by default in modern drivers

If you're running MongoDB driver version 3.6 or later (and you should be), retryable writes are enabled by default in most languages: Node.js, Python, Java, Go, C#, and others. The same goes for retryable reads in version 4.2+. You don't need to flip any flags or write boilerplate code to get this protection.

However, "on by default" doesn't mean universal. Retryable operations require specific conditions to be met:

• A replica set or sharded cluster deployment (not standalone servers)
• Explicit acknowledgment of writes (which is the default, but worth verifying)
• No multi-document transactions in progress at the moment of the error
• No session associated with a failed operation (for retryable reads)

In development with a local MongoDB instance running as a single node, retryable operations won't activate. This is a common source of confusion: code works reliably locally but encounters transient failures in staging. Always test retry behavior against a replica set, even if it's just three containers on your laptop.

To confirm retryable writes are active in Node.js with the native driver:

const { MongoClient } = require('mongodb');

const client = new MongoClient(uri, {
  retryWrites: true,  // explicitly set (default since 4.2)
  w: 'majority'       // acknowledgment required for retryable writes
});

const db = client.db('myapp');
const collection = db.collection('users');

// This operation is now retryable on transient network errors
await collection.insertOne({ name: 'Alice', email: 'alice@example.com' });

If your URI connection string includes retryWrites=false, you're opting out. Double-check that your deployment string doesn't accidentally disable this safety net.

Single retry for supported operations

Here's a critical detail: the driver retries exactly once. Not zero times, not three times—one retry. If the first attempt fails with a retryable error (like a network timeout or a brief replica set failover), the driver will issue the command again. If it fails a second time, that error propagates to your application code.

The key insight is that not all operations are retryable. Write operations that don't change state (like find, aggregate, or count) are retryable out of the box. Write operations that change state (like insertOne, updateOne, deleteOne, and replaceOne) are also retryable, but only if you meet the conditions mentioned above. Complex operations like insertMany with mixed successes are not retryable.

This is where the single-retry model becomes important: the driver can't know whether a failed write actually succeeded on the server before the connection dropped. That's why idempotency becomes your responsibility.

Consider a real scenario: you insert a document, the write succeeds on the server, but the network dies before the driver receives the acknowledgment. The driver retries. Now you risk inserting the same document twice unless you have a unique index or explicit idempotency logic in place.

Writes must be idempotent-safe

Idempotency is non-negotiable when working with retryable writes. An idempotent operation produces the same result whether it runs once or multiple times. For MongoDB writes, this typically means either:

1. Using a unique constraint (like a unique index on a field you control)
2. Using upsert with a specific query that ensures one-to-one mapping between input and database state
3. Designing your operation so running it twice is harmless

Let's look at a practical example from an e-commerce context. Suppose you're creating an order after payment succeeds. The payment system calls your API, which inserts an order document and charges the customer. If the insert succeeds but the response is lost:

// Vulnerable to double-processing without idempotency
const createOrder = async (collection, customerId, items, paymentId) => {
  // If this insert succeeds but the network fails, a retry will create a duplicate
  const result = await collection.insertOne({
    customerId,
    items,
    paymentId,
    createdAt: new Date(),
    status: 'pending'
  });
  return result.insertedId;
};

A better approach uses the payment ID as a unique identifier:

// Idempotent via unique constraint
const createOrder = async (collection, customerId, items, paymentId) => {
  try {
    // Ensure paymentId is unique across all orders
    const result = await collection.insertOne({
      customerId,
      items,
      paymentId,  // This field must have a unique index
      createdAt: new Date(),
      status: 'pending'
    });
    return result.insertedId;
  } catch (err) {
    if (err.code === 11000) {
      // Duplicate key error: this order already exists
      // Retrieve and return the existing order
      const existing = await collection.findOne({ paymentId });
      return existing._id;
    }
    throw err;
  }
};

Make sure the unique index actually exists:

// Create this once, during application startup
await collection.createIndex({ paymentId: 1 }, { unique: true });

Another pattern is upsert-based idempotency, common in update operations. If you're updating a user's profile:

// Idempotent upsert pattern
const updateUserProfile = async (collection, userId, profileData) => {
  const result = await collection.updateOne(
    { _id: userId },  // Specific, unique query
    { $set: profileData },
    { upsert: false }  // Only update, don't insert
  );
  return result;
};

The specificity of the filter is what makes this safe. Even if the operation retries, the same document gets updated with the same data.

Still handle errors in application code

Automatic retries are a safety net, not a complete solution. Your application code must still handle errors gracefully. Several categories of errors bypass the retry mechanism:

• Permanent errors: Authentication failures, invalid commands, constraint violations (unless they're from idempotency checks)
• Non-retryable network errors: DNS resolution failures, connection refused
• Errors after the second attempt: The retry already happened and failed again
• Multi-statement transactions: Retryable writes don't apply within explicit sessions with transactions

A robust error-handling pattern in production looks like this:

const insertUserWithFallback = async (collection, userData) => {
  try {
    const result = await collection.insertOne(userData);
    return { success: true, id: result.insertedId };
  } catch (err) {
    if (err.code === 11000) {
      // Duplicate key: idempotency worked, but the resource already exists
      console.warn('User already exists, returning existing record');
      const existing = await collection.findOne({ email: userData.email });
      return { success: true, id: existing._id, duplicated: true };
    }

    if (err.name === 'MongoNetworkTimeoutError') {
      // Network error that wasn't retried successfully
      console.error('Network timeout after retries, alerting ops team');
      // Send to error tracking, return graceful failure
      return { success: false, code: 'NETWORK_TIMEOUT' };
    }

    if (err.name === 'MongoWriteConcernError') {
      // Write concern failed
      console.error('Write concern error:', err.message);
      return { success: false, code: 'WRITE_CONCERN_FAILED' };
    }

    // Unexpected error
    console.error('Unexpected error inserting user:', err);
    throw err;
  }
};

Notice we're not retrying manually here. The driver already did that. Instead, we're classifying the final error and deciding how to respond. For transient issues, you might queue the operation for later. For permanent issues, you might fail fast.

Common mistakes

Mistake 1: Assuming retryable writes work on a standalone server. They don't. A single-node MongoDB instance has no replica set, so retryable writes are disabled. Always develop against a replica set topology, even locally. Use Docker Compose to spin up a three-node replica set in seconds.

Mistake 2: Ignoring idempotency in update operations. Even though updateOne looks safe (it's only touching one document), a network failure before acknowledgment can cause confusion. Ensure your update filters are specific enough that applying the same update twice is harmless, or use unique constraints.

Mistake 3: Implementing manual retry logic on top of automatic retries. This doubles the retry count and complicates error handling. Trust the driver's built-in mechanism and only add retry logic at higher layers (like a job queue) for operations that fail after the automatic retry.

Mistake 4: Not testing retry behavior in staging. A change that works fine locally might fail under load in production if you never verified retryable writes against a real replica set with actual failover.

Mistake 5: Using bulk write operations and expecting all-or-nothing retryability. bulkWrite and insertMany are not retryable if they encounter errors partway through. Design your bulk operations to be idempotent on a per-document basis instead.

Key takeaways

• Retryable writes and reads are enabled by default in modern MongoDB drivers but require a replica set deployment and proper write concern settings.
• The driver retries exactly once on transient errors. Your application must still handle errors that fail after the retry.
• Idempotency is essential: use unique constraints, upserts with specific filters, or payment/request IDs to ensure that a retried write doesn't create duplicates.
• Don't layer manual retries on top of automatic ones. Classify errors and handle them appropriately in your application code.
• Test retry behavior against a real replica set, not a standalone server, to catch idempotency and error-handling bugs before production.
• Complex operations like bulkWrite are not retryable, so design them to be idempotent at the document level.