Bulk writes in MongoDB in MongoDB, NoSQL & Data Modeling

Bulk writes in MongoDB in MongoDB, NoSQL & Data Modeling

When you're inserting, updating, or deleting thousands of documents in MongoDB, firing individual operations will crush your throughput and waste network resources. Bulk writes let us combine multiple operations into a single round trip, dramatically reducing latency overhead and improving performance by an order of magnitude.

Understanding bulkWrite with mixed operations

MongoDB's bulkWrite() method accepts an array of operation objects, each specifying an action like insert, update, or delete. This is where the real power lies: we can mix different operation types in a single batch and execute them all together, rather than making separate calls for each operation type.

Consider a real scenario where we're syncing customer data from an upstream system. Some customers are new (insert), some have updated details (update), and some have churned (delete). With individual operations, we'd make three separate database calls per customer. With bulkWrite, we combine them all:

const operations = [
  {
    insertOne: {
      document: {
        customerId: 'cust_9001',
        name: 'Alice Chen',
        email: 'alice@example.com',
        createdAt: new Date()
      }
    }
  },
  {
    updateOne: {
      filter: { customerId: 'cust_8542' },
      update: {
        $set: {
          email: 'newemail@example.com',
          lastUpdated: new Date()
        }
      }
    }
  },
  {
    deleteOne: {
      filter: { customerId: 'cust_7821' }
    }
  },
  {
    updateMany: {
      filter: { status: 'inactive', lastLogin: { $lt: new Date(Date.now() - 90 * 24 * 60 * 60 * 1000) } },
      update: { $set: { archived: true } }
    }
  }
];

const result = await collection.bulkWrite(operations);
console.log(`Inserted: ${result.insertedCount}, Modified: ${result.modifiedCount}, Deleted: ${result.deletedCount}`);

The bulkWrite() method returns a BulkWriteResult object with counts for each operation type, giving us visibility into what actually happened. This single call handles four different operation types in one network round trip.

ordered: false for speed

By default, bulkWrite() executes operations in order and stops on the first error. This is safe but slow. For most data-sync scenarios, we don't care about order—we just want maximum throughput. Setting ordered: false tells MongoDB to execute all operations in parallel, skipping failures and continuing with the rest.

The speed difference is significant. Imagine syncing 10,000 product records where 50 have constraint violations. With ordered execution, we'd hit the first error and stop, wasting the opportunity to process the remaining 9,950 records. With unordered execution, MongoDB runs operations in parallel and collects all failures for us to handle afterward.

const operations = items.map(item => ({
  updateOne: {
    filter: { productId: item.productId },
    update: { $set: { price: item.newPrice, stock: item.quantity } },
    upsert: true
  }
}));

const result = await collection.bulkWrite(operations, { ordered: false });

console.log(`Modified: ${result.modifiedCount}, Upserted: ${result.upsertedCount}`);
if (result.writeErrors && result.writeErrors.length > 0) {
  console.log(`Write errors: ${result.writeErrors.length}`);
  result.writeErrors.forEach(err => {
    console.log(`Index ${err.index}: ${err.errmsg}`);
  });
}

We still get the error details in the result object, so we haven't lost visibility—we've just prioritized speed. For a bulk sync job, this is exactly what we want. The operations that fail are typically the minority, and we can handle them in a second pass or log them for manual review.

Batch in chunks of a few thousand

It's tempting to dump all 100,000 records into a single bulkWrite call. Don't. MongoDB has practical limits on request size and memory, and sending enormous batches can actually slow things down and risk timeouts. We batch our operations into chunks of 2,000 to 5,000 and execute them sequentially. This keeps memory pressure reasonable and allows early error feedback.

Here's a pattern we use for large syncs in production:

async function bulkSyncRecords(records, collection, batchSize = 3000) {
  const totalBatches = Math.ceil(records.length / batchSize);
  let successCount = 0;
  let errorCount = 0;

  for (let i = 0; i < records.length; i += batchSize) {
    const batch = records.slice(i, i + batchSize);
    const operations = batch.map(record => ({
      updateOne: {
        filter: { externalId: record.id },
        update: { $set: record },
        upsert: true
      }
    }));

    try {
      const result = await collection.bulkWrite(operations, { ordered: false });
      successCount += result.modifiedCount + result.upsertedCount;
      
      if (result.writeErrors && result.writeErrors.length > 0) {
        errorCount += result.writeErrors.length;
        console.warn(`Batch ${Math.floor(i / batchSize) + 1}/${totalBatches}: ${result.writeErrors.length} errors`);
      }
    } catch (err) {
      console.error(`Batch ${Math.floor(i / batchSize) + 1}/${totalBatches} failed:`, err.message);
      errorCount += batch.length;
    }
  }

  return { successCount, errorCount, total: records.length };
}

This approach gives us progress visibility and prevents one oversized batch from blocking the entire sync. If a batch fails, we lose that batch's worth of data but not the entire sync. We've found that 3,000–5,000 operations per batch strikes a good balance between throughput and safety on most cloud-hosted MongoDB instances.

Handle partial failures

With ordered: false, some operations will fail while others succeed. This is by design, but we must handle it properly. The result object contains writeErrors (individual operation failures) and writeConcernError (whole-batch issues). We need to inspect both and decide what to do with failed records.

In a typical e-commerce scenario, we're syncing orders. A few might fail due to duplicate key constraints or validation errors. We want to log those, notify a team member if there are many, and continue processing. Here's a realistic handler:

async function logAndRetryFailures(result, failedRecords, collection) {
  if (result.writeErrors && result.writeErrors.length > 0) {
    const retryOps = [];
    
    for (const error of result.writeErrors) {
      const failedRecord = failedRecords[error.index];
      console.error(`Operation ${error.index} failed: ${error.errmsg}`, failedRecord);

      // Retry with explicit upsert strategy
      if (error.code === 11000) {
        // Duplicate key: update instead of insert
        retryOps.push({
          updateOne: {
            filter: { _id: failedRecord._id },
            update: { $set: failedRecord },
            upsert: true
          }
        });
      } else if (error.code === 121) {
        // Document validation error: log for manual review
        await logToFailureQueue(failedRecord, error.errmsg);
      } else {
        // Other error: queue for retry
        await logToFailureQueue(failedRecord, error.errmsg);
      }
    }

    // Retry the fixable ones
    if (retryOps.length > 0) {
      const retryResult = await collection.bulkWrite(retryOps, { ordered: false });
      return { originalFailed: result.writeErrors.length, retrySucceeded: retryResult.modifiedCount };
    }
  }

  return { originalFailed: 0 };
}

The key insight is that error codes tell us why an operation failed. Code 11000 is a duplicate key—we can often retry differently. Code 121 is a schema validation failure—we should log it for investigation. By inspecting error codes, we can automate recovery for common issues and reduce manual intervention.

Common mistakes

• Not using bulkWrite at all. Teams sometimes loop over individual updates instead of batching. This multiplies network latency and kills throughput. Always batch when processing more than a handful of documents.
• Batches that are too large. Sending 50,000 operations in one call can timeout or exhaust memory. Start with 3,000–5,000 and adjust based on document size and network conditions.
• Ignoring ordered: false. Leaving ordered as true (the default) for performance-critical syncs defeats the purpose. Use ordered: true only when operation order genuinely matters.
• Not checking result errors. A successful bulkWrite call doesn't mean all operations succeeded. Always inspect result.writeErrors and result.writeConcernError.
• Combining bulkWrite with transactions carelessly. Bulk writes work within transactions but can lock resources. For very large batches, unordered bulk writes without transactions often perform better.

Key takeaways

• Use bulkWrite() to combine insert, update, and delete operations into a single network call, reducing latency overhead and multiplying throughput.
• Set ordered: false for data-sync workloads to enable parallel execution and skip failures without stopping the entire batch.
• Chunk large datasets into batches of 2,000–5,000 operations to balance memory pressure and performance; avoid dumping tens of thousands into one call.
• Always inspect writeErrors in the result object. Classify failures by error code and decide whether to retry, log, or escalate.
• Bulk writes are essential for keeping insertion and update throughput high in production; individual operations don't scale.