Handling large documents: a hands-on guide for engineers

Why Document Size Matters in MongoDB

MongoDB enforces a hard limit: documents cannot exceed 16 MB. Sounds generous until you're managing a user profile that accumulates years of order history, or an event log that grows unbounded. Handling large documents: a hands-on guide for engineers is essential because undersizing your approach leads to query sprawl, replication lag, and production incidents at 3 AM. We've all seen schemas that worked beautifully at launch but became nightmares as data accumulated. The good news: there are proven patterns to keep documents lean and performant.

Split Growing Data into Child Documents

When a document naturally contains a one-to-many relationship that grows over time, denormalizing everything into an array becomes problematic. Consider an e-commerce order document that stores line items, then shipments, then tracking updates, then customer notes. Within six months, you've got a bloated monster.

The solution: use referenced or embedded child documents with a parent-child pattern. Store the core order data in one collection, and orders-events or orders-lineItems in another, linked by order ID.

// Before: single bloated document
{
  _id: ObjectId("..."),
  customerId: 123,
  orderDate: ISODate("2024-01-15"),
  lineItems: [
    { sku: "ABC-001", qty: 2, price: 49.99 },
    { sku: "ABC-002", qty: 1, price: 79.99 }
    // ... grows indefinitely
  ],
  shipments: [
    { trackingNumber: "1Z...", carrier: "UPS", status: "delivered" },
    // ... history
  ],
  events: [
    { type: "created", timestamp: ISODate("...") },
    { type: "paid", timestamp: ISODate("...") },
    { type: "shipped", timestamp: ISODate("...") },
    // ... audit trail grows forever
  ]
}

// After: structured relationships
// Collection: orders
{
  _id: ObjectId("..."),
  customerId: 123,
  orderDate: ISODate("2024-01-15"),
  total: 129.98,
  status: "delivered"
}

// Collection: orders_lineItems
{
  _id: ObjectId("..."),
  orderId: ObjectId("..."),
  sku: "ABC-001",
  qty: 2,
  price: 49.99
}

// Collection: orders_events
{
  _id: ObjectId("..."),
  orderId: ObjectId("..."),
  type: "shipped",
  timestamp: ISODate("..."),
  metadata: { carrier: "UPS", trackingNumber: "1Z..." }
}

This approach scales naturally. A typical order document stays under 1 KB; events and items live elsewhere. When you need the full order context, you query the parent and perform lookups or aggregation pipeline stages with $lookup. Yes, it requires multiple queries, but the performance gain from scanning smaller, cacheable documents outweighs the extra I/O in most scenarios.

Move Binary Data to Object Storage

Never embed large binary data—PDFs, images, videos—directly into MongoDB documents. A single invoice PDF can be 2–5 MB; storing ten of them per customer document kills your collection instantly. Even when there's only one attachment per document, you're bloating queries that don't need that binary data.

Instead, store binary content in S3, Azure Blob Storage, or Google Cloud Storage, and keep only metadata and a reference in MongoDB.

// Before: embedding binary
{
  _id: ObjectId("..."),
  customerId: 123,
  invoiceNumber: "INV-2024-001",
  invoiceData: BinData(0, "JVBERi0xLjQK...")  // 3 MB PDF embedded
}

// After: reference + metadata
{
  _id: ObjectId("..."),
  customerId: 123,
  invoiceNumber: "INV-2024-001",
  invoiceUrl: "s3://my-bucket/invoices/2024/001/INV-2024-001.pdf",
  invoiceSize: 3145728,
  uploadedAt: ISODate("2024-01-20"),
  contentType: "application/pdf"
}

The document stays under 1 KB. When your application needs the PDF, it retrieves the URL and fetches the file directly from object storage, which is optimized for large binary retrieval. Your MongoDB queries are snappy, replication is efficient, and you can scale storage independently.

Project Only Needed Fields

Even if your documents are reasonably sized, querying unnecessary fields adds latency and network overhead. In a sharded cluster, this multiplies across nodes. Always use projection to retrieve only what you need.

// Inefficient: fetches entire document
db.users.find({ status: "active" });

// Better: project only required fields
db.users.find(
  { status: "active" },
  { _id: 1, email: 1, name: 1, lastLogin: 1 }
);

// In aggregation pipelines, project early to reduce downstream data
db.orders.aggregate([
  { $match: { status: "pending" } },
  { $project: { _id: 1, customerId: 1, total: 1, orderDate: 1 } },
  { $group: { _id: "$customerId", totalSpent: { $sum: "$total" } } }
]);

This is especially critical for high-volume APIs. If you're returning 10,000 documents to serialize to JSON, an extra 5 KB per document means 50 MB of unnecessary data crossing the network. Projection reduces memory pressure on both server and client, lowers bandwidth, and improves cache hit rates.

Monitor Average Document Size

You can't optimize what you don't measure. MongoDB provides tools to inspect collection statistics, including average document size. Set up regular monitoring to catch bloat before it becomes a crisis.

// Get collection stats
db.users.stats();

// Output includes:
// {
//   "count": 50000,
//   "size": 52428800,
//   "avgObjSize": 1048.576
// }

// In aggregation, calculate document size dynamically
db.orders.aggregate([
  { $project: { size: { $bsonSize: "$$ROOT" } } },
  { $group: { _id: null, avgSize: { $avg: "$size" }, maxSize: { $max: "$size" } } }
]);

// Output:
// { _id: null, avgSize: 2048, maxSize: 15728640 }

If your average order document is 2 MB and growing 10% per month, you have a data model problem. Set alerts at thresholds—say, 5 MB average or any document over 10 MB—so your team addresses it during sprint planning, not during a production outage.

Integrate these metrics into your observability stack. Tools like MongoDB Atlas provide built-in monitoring; for self-hosted, use prometheus exporters or custom scripts that run weekly.

Common Mistakes

• Deferring the split until too late: When a collection reaches millions of documents with an average size of 8 MB, refactoring is painful. Identify growth patterns early and proactively separate concerns.
• Over-normalizing and killing read performance: Not every one-to-many relationship demands a separate collection. If your child documents are small (under 100 bytes) and queries always need them together, embedding is fine. Use your judgment based on access patterns.
• Forgetting to index foreign keys: If you split data into child collections, ensure indexes on the linking fields (orderId, customerId, etc.). A lookup without an index defeats the purpose of the split.
• Storing file paths instead of URLs: When moving binary data to object storage, don't store local file system paths. Store full, immutable URLs so clients can fetch files directly without your app as a middleman.
• Ignoring GridFS for medium-sized binaries: For files 5–10 MB that you want to keep within MongoDB (not object storage), GridFS is purpose-built. It splits binaries across multiple documents and handles reassembly transparently.

Key Takeaways

• MongoDB's 16 MB limit is real; design for growth from day one.
• Move growing one-to-many data into child collections linked by parent ID; queries scale better with smaller, focused documents.
• Store binary files in object storage and reference them in MongoDB; your documents stay lean and efficient.
• Always project only the fields you need; it cuts latency, bandwidth, and memory pressure across the stack.
• Monitor average document size and set alerts; catching bloat early prevents production firefighting.
• Balance normalization against access patterns; not every split improves performance, so profile your workload.