Aggregation pipeline basics: what developers should know

Aggregation pipeline basics: what developers should know

MongoDB's aggregation pipeline is one of the most powerful features for transforming and analyzing data inside the database itself, rather than pulling raw records into your application layer. If you're working with production data—whether it's e-commerce orders, user activity logs, or financial transactions—you'll quickly find that simple find() queries don't cut it. The aggregation pipeline lets you filter, group, reshape, and compute metrics in a single pass, making your queries both faster and cleaner to reason about.

In this article, we'll walk through the fundamental stages that make up the aggregation pipeline, show you how to think about performance, and share the debugging techniques we use when building complex pipelines in production systems.

How the aggregation pipeline works

Think of the aggregation pipeline as a series of processing stations on an assembly line. Each document enters the first stage, gets transformed or filtered, and then passes to the next stage. The output of one stage becomes the input to the next. This is fundamentally different from a traditional query: instead of MongoDB just finding documents, it's actively reshaping and computing across them.

Here's a simple mental model:

• $match — Filter documents (like a WHERE clause)
• $group — Aggregate and group results (like GROUP BY)
• $sort — Order the output
• $project — Shape the output fields
• $limit — Trim results

Each stage processes every document that flows through it, and the order matters. A document rejected by $match never reaches $group, which is crucial for performance.

Common stages: building blocks of your pipeline

Let's explore the stages you'll use most often. We'll use a real scenario: an e-commerce database with orders, each containing a user ID, total amount, status, and creation date.

$match — Filter early

$match is your first line of defense. It filters documents before any heavy processing happens. Without it, downstream stages waste resources processing irrelevant data.

db.orders.aggregate([
  {
    $match: {
      status: "completed",
      createdAt: { $gte: new Date("2024-01-01") }
    }
  }
])

This stage filters to only completed orders from January 2024 onwards. Any order that doesn't match is dropped immediately.

$group — Aggregate and compute

After filtering, you typically aggregate. $group lets you compute sums, averages, counts, and more across matching documents. You define a _id field (what you're grouping by) and accumulator operators.

db.orders.aggregate([
  {
    $match: {
      status: "completed",
      createdAt: { $gte: new Date("2024-01-01") }
    }
  },
  {
    $group: {
      _id: "$userId",
      totalSpent: { $sum: "$amount" },
      orderCount: { $sum: 1 },
      avgOrderValue: { $avg: "$amount" }
    }
  }
])

Now you have one document per user showing their total spending, number of orders, and average order value. The _id field is "$userId" because we're grouping by user.

$sort — Order results

After grouping, you often want to sort by one of your computed fields. $sort expects a direction: 1 for ascending, -1 for descending.

db.orders.aggregate([
  {
    $match: {
      status: "completed",
      createdAt: { $gte: new Date("2024-01-01") }
    }
  },
  {
    $group: {
      _id: "$userId",
      totalSpent: { $sum: "$amount" },
      orderCount: { $sum: 1 }
    }
  },
  {
    $sort: { totalSpent: -1 }
  }
])

This returns your top spenders first. The $sort stage respects indexes if it appears early in the pipeline (more on this later).

$project — Shape the output

$project lets you include, exclude, or rename fields. It's useful for cleaning up output and hiding internal fields before sending data to the client.

db.orders.aggregate([
  {
    $match: {
      status: "completed",
      createdAt: { $gte: new Date("2024-01-01") }
    }
  },
  {
    $group: {
      _id: "$userId",
      totalSpent: { $sum: "$amount" },
      orderCount: { $sum: 1 }
    }
  },
  {
    $sort: { totalSpent: -1 }
  },
  {
    $project: {
      userId: "$_id",
      totalSpent: 1,
      orderCount: 1,
      _id: 0
    }
  }
])

Now the output field is "userId" instead of "_id", and we've explicitly hidden the _id field by setting it to 0.

$limit — Stop early

$limit caps the number of documents returned. Combined with $sort, it gives you pagination.

db.orders.aggregate([
  {
    $match: {
      status: "completed",
      createdAt: { $gte: new Date("2024-01-01") }
    }
  },
  {
    $group: {
      _id: "$userId",
      totalSpent: { $sum: "$amount" }
    }
  },
  {
    $sort: { totalSpent: -1 }
  },
  {
    $limit: 10
  }
])

This returns only the top 10 spenders. In production, we often use $limit right after $sort to prevent memory bloat when processing millions of documents.

Match first: the golden rule of performance

The single most important principle for aggregation pipeline performance is filter early. Every document you remove at the $match stage is one fewer document that downstream stages have to process, sort, or group.

Consider this pipeline:

db.orders.aggregate([
  {
    $group: {
      _id: "$userId",
      totalSpent: { $sum: "$amount" }
    }
  },
  {
    $match: {
      totalSpent: { $gt: 1000 }
    }
  }
])

This groups all orders (potentially millions) before filtering. It's wasteful. Rearrange it:

db.orders.aggregate([
  {
    $match: {
      status: "completed"
    }
  },
  {
    $group: {
      _id: "$userId",
      totalSpent: { $sum: "$amount" }
    }
  },
  {
    $match: {
      totalSpent: { $gt: 1000 }
    }
  }
])

You can have multiple $match stages. The first one uses an index (if available) to reduce the dataset. The second filters after grouping. MongoDB's query optimizer is smart, but being explicit pays dividends.

Use indexes: make the pipeline lightning fast

Indexes are just as critical for aggregation pipelines as they are for regular queries. A $match stage at the beginning of your pipeline can use an index to quickly locate documents. Similarly, $sort can use an index if it appears early and no blocking stages come before it.

Example: if you have an index on { status: 1, createdAt: 1 }, this pipeline will be fast:

db.orders.aggregate([
  {
    $match: {
      status: "completed",
      createdAt: { $gte: new Date("2024-01-01") }
    }
  },
  {
    $sort: { totalAmount: -1 }
  },
  {
    $group: {
      _id: "$userId",
      totalSpent: { $sum: "$amount" }
    }
  }
])

MongoDB will use the index on status and createdAt to filter quickly. Once you hit $group, you lose index benefits—but by then you've already eliminated most documents.

Use explain() to see if your pipeline is using indexes:

db.orders.aggregate([...], { explain: true })

Look for "executionStages" in the output. If you see "COLLSCAN" early on, you're missing an index.

Test in Compass: visualize each stage

MongoDB Compass, the official GUI, has an aggregation pipeline builder that's invaluable during development. You can write each stage, see the output in real-time, and watch how document shape changes stage by stage.

Open Compass, navigate to your collection, click the "Aggregation" tab, and add stages. As you type each stage, Compass shows sample output below. This immediate feedback loop is much faster than trial-and-error in your application code.

We also recommend starting with small datasets or using $limit during development, then removing it in production. Compass will show you exactly what each stage produces, making debugging vastly easier.

Common mistakes

Filtering after grouping unnecessarily: If you can filter before $group, do it. Grouping is expensive.

Forgetting that $group reduces documents: After $group, your document count typically drops dramatically. Don't expect post-group stages to process millions of documents.

Using $match on computed fields: You can't use $match on fields created by $project or $group in an earlier stage of the same pipeline. Use the computed field name after it's created, but remember the context has changed.

Over-relying on the pipeline for application logic: Aggregation is powerful, but if your pipeline becomes a 15-stage monster, consider whether some logic belongs in your application code instead. A complex pipeline is hard to test, version, and debug.

Not using allowDiskUse: By default, MongoDB holds the entire result set in memory. For large aggregations, add { allowDiskUse: true } as an option. It's slower but won't crash:

db.orders.aggregate([...], { allowDiskUse: true })

Key takeaways

• The aggregation pipeline processes documents through sequential stages; each stage's output feeds the next.
• Always place $match early to filter documents before expensive operations like $group.
• Use indexes on $match and early $sort stages to gain massive speed improvements.
• MongoDB Compass's aggregation builder is your best friend during development; use it to visualize output at each stage.
• Avoid filtering on computed fields in the same $match stage; use a second $match stage instead.
• Keep pipelines readable; if you exceed 10 stages regularly, consider pushing some logic to your application.