Aggregation performance tips: what developers should know

Aggregation performance tips: what developers should know

MongoDB aggregations are powerful, but they're also easy to get wrong at scale. A pipeline that works fine with 10,000 documents can grind to a halt at 10 million. The difference between a fast aggregation and a slow one often comes down to a few simple decisions made early in the pipeline design. We'll walk through the practical strategies that keep pipelines fast at scale, drawing from real production scenarios where these patterns have made the difference between responsive APIs and timeouts.

Match and sort early — Let the first stages use indexes

The most impactful optimization is placing $match stages as early as possible in your pipeline. MongoDB can push down a $match at the start of a pipeline to the query planner, which means it can use indexes to reduce the working set before any other processing happens. This is where index strategy meets aggregation design.

Consider a common e-commerce scenario: you need to find all orders placed in the last 30 days, group by customer, and calculate average order value. A naive pipeline might load all orders, then filter. A smart one filters first.

// ❌ Slow: filters after other stages
db.orders.aggregate([
  {
    $group: {
      _id: "$customerId",
      avgValue: { $avg: "$amount" },
      count: { $sum: 1 }
    }
  },
  {
    $match: {
      createdAt: { $gte: new Date(Date.now() - 30 * 24 * 60 * 60 * 1000) }
    }
  }
])

// ✅ Fast: filters first, uses index
db.orders.aggregate([
  {
    $match: {
      createdAt: { $gte: new Date(Date.now() - 30 * 24 * 60 * 60 * 1000) }
    }
  },
  {
    $group: {
      _id: "$customerId",
      avgValue: { $avg: "$amount" },
      count: { $sum: 1 }
    }
  }
])

With an index on createdAt, the second version will only process relevant documents. The first version processes millions, then throws most away. In production, we've seen this simple reordering drop execution time from 8 seconds to 200 milliseconds on a 50-million-document collection.

Sort order matters too. If you need to sort before a $limit, do it early. MongoDB can use an index to satisfy a sort without materializing all results in memory.

// Early sort + limit can use index
db.orders.aggregate([
  {
    $match: { status: "completed" }
  },
  {
    $sort: { createdAt: -1 }
  },
  {
    $limit: 20
  }
])

Check your indexes. On orders, you'd want a compound index on {status: 1, createdAt: -1} to cover both the match and sort. Use explain() to verify that stages are using indexes instead of scanning collections.

Project early — Drop unneeded fields to reduce memory

Every field you carry through your pipeline consumes memory. Once data flows into stages like $group or $sort, MongoDB builds in-memory structures. Smaller documents mean fewer memory allocations, faster processing, and less pressure on the working set.

Add a $project stage right after your initial $match to keep only what you need downstream. This is especially critical before grouping or joining.

// ❌ Carrying unnecessary fields through the pipeline
db.orders.aggregate([
  {
    $match: { status: "completed" }
  },
  {
    $group: {
      _id: "$customerId",
      total: { $sum: "$amount" }
    }
  }
  // But we loaded userId, email, phone, address, notes, etc.
])

// ✅ Project only what's needed
db.orders.aggregate([
  {
    $match: { status: "completed" }
  },
  {
    $project: {
      customerId: 1,
      amount: 1
    }
  },
  {
    $group: {
      _id: "$customerId",
      total: { $sum: "$amount" }
    }
  }
])

If your documents are large (embedded arrays, long text fields, binary data), the difference is dramatic. We once debugged a report pipeline that was hitting disk and running for 3 minutes. A single $project to drop a 500 KB array that wasn't used until the final stage brought it under 2 seconds.

You can also use $project to compute derived fields and simplify logic downstream:

db.orders.aggregate([
  {
    $match: { status: "completed" }
  },
  {
    $project: {
      customerId: 1,
      amount: 1,
      year: { $year: "$createdAt" },
      month: { $month: "$createdAt" }
    }
  },
  {
    $group: {
      _id: { year: "$year", month: "$month" },
      revenue: { $sum: "$amount" }
    }
  }
])

This approach is cleaner and faster than computing year and month inside the $group stage.

Avoid large $unwind — It multiplies documents

The $unwind stage can be a silent killer. It takes one document with an array and produces N documents, one for each array element. If you're unwinding a large array across millions of documents, the intermediate result set explodes.

Imagine a product catalog where each product has 1,000 reviews embedded:

// ❌ Dangerous: 1M products × 1000 reviews = 1B documents in memory
db.products.aggregate([
  {
    $unwind: "$reviews"
  },
  {
    $group: {
      _id: "$_id",
      avgRating: { $avg: "$reviews.rating" }
    }
  }
])

You go from 1 million documents to 1 billion in the pipeline. This will almost certainly spill to disk, and disk I/O will dominate your runtime.

Better approaches depend on your use case:

• If reviews are truly embedded: Use array operators like $avg on the array directly, or use $reduce to compute within a single document.
• If reviews should be a separate collection: Restructure your schema. Reviews belong in their own collection with a foreign key to products. Join with $lookup if needed, but don't embed large arrays.
• If you must unwind: Filter before unwinding to reduce array size first.

// ✅ Better: compute on array without unwinding
db.products.aggregate([
  {
    $project: {
      _id: 1,
      name: 1,
      avgRating: { $avg: "$reviews.rating" },
      reviewCount: { $size: "$reviews" }
    }
  }
])

// ✅ Or filter first if you must unwind
db.products.aggregate([
  {
    $match: { "reviews.rating": { $gte: 4 } }
  },
  {
    $unwind: "$reviews"
  },
  {
    $match: { "reviews.rating": { $gte: 4 } }
  },
  {
    $group: {
      _id: "$_id",
      highRatingCount: { $sum: 1 }
    }
  }
])

If you find yourself unwinding regularly, it's often a sign to reconsider your data model. Embedding works well for small, read-only, or slowly-growing arrays. Large or frequently-queried arrays should be separate.

allowDiskUse — Only for offline jobs — fix the pipeline for APIs

MongoDB has an allowDiskUse option that spills aggregation results to disk when memory limits are hit. It's tempting to enable it globally as a safety valve. Don't.

allowDiskUse is for batch jobs and reports that run at night, not for API requests. Disk I/O is 100x slower than memory. A query that takes 500 milliseconds in memory might take 10 seconds with disk spillage. Your API timeout fires before results return.

// ❌ For an API: this will be slow
db.orders.aggregate(pipeline, { allowDiskUse: true })

// ✅ For an overnight batch report: fine
db.orders.aggregate(pipeline, { allowDiskUse: true })

If you're hitting memory limits, the real fix is the pipeline itself. Revisit:

• Are you matching early to reduce the working set?
• Are you projecting to drop unused fields?
• Are you unwinding large arrays?
• Are you grouping on high-cardinality fields (like user IDs) without a prior match?

A well-optimized pipeline stays in memory and completes fast. If you enable allowDiskUse for an API, you're masking the real problem and delivering poor user experience.

For background jobs where speed is less critical but reliability is paramount, allowDiskUse is appropriate. Just not for user-facing requests.

Common mistakes

• Matching late: Filtering after expensive operations like $group or $lookup defeats indexing. Always move $match stages forward.
• Carrying every field: When in doubt, project early. You can always access omitted fields if you change your mind, but carrying garbage through the pipeline wastes memory and time.
• Assuming allowDiskUse solves memory issues: It doesn't. It trades speed for reliability. Fix the pipeline instead.
• Unwinding without filtering: Before unwinding arrays, filter if possible. Even a small reduction in array size or document count has exponential impact downstream.
• Ignoring explain output: Always run explain() on production-scale data. A pipeline that's fast on a dev database with 10K records can be a disaster at real scale. Look for COLLSCAN stages and large memory usage.

Key takeaways

• Place $match early to leverage indexes and reduce the working set before other processing.
• Project immediately after matching to drop unused fields and reduce memory overhead.
• Avoid large $unwind operations; they explode document counts. Use array operators or restructure your schema.
• allowDiskUse is for batch jobs only, never for APIs. Slow disk spillage defeats the purpose of a responsive API.
• Use explain() to verify that your pipeline uses indexes and doesn't materialize unnecessary data.
• Think of the aggregation pipeline as a production line: filter and shape early, compute late, and never carry waste forward.