100 days of MongoDB and NoSQL: the recap for production

100 Days of MongoDB and NoSQL: The Recap for Production

We've spent the last 100 days building mental models, running experiments, and learning from production incidents. 100 days of MongoDB and NoSQL: the recap for production distills the lessons that actually matter when you're shipping systems at scale. If you've been following along, you've seen how theory breaks down in the real world—and where it holds up. This recap is what we'd tell the team on day one if we could start over.

Model for Your Queries — Embed What You Read Together; Reference What Grows

The single most expensive mistake we see in production is a schema designed around normalization instead of access patterns. You're not building a relational database. MongoDB rewards you for thinking about how data moves through your application, not how it's logically separate.

Start with this rule: embed what you read together, reference what grows. If you always fetch a user's profile, avatar, and preferences in the same request, embed them in one document. The read is atomic, the network round trip disappears, and your application code stays clean.

But if that user has thousands of orders, or thousands of followers, you reference instead. Embedding unbounded arrays bloats your documents, slows queries, and creates maintenance nightmares when you need to update a single item in the array.

Here's what this looks like in practice. For an e-commerce platform, a user document might look like this:

{
  _id: ObjectId("..."),
  username: "alice",
  email: "alice@example.com",
  profile: {
    displayName: "Alice",
    avatar: "https://...",
    bio: "...",
    joinedAt: ISODate("2023-01-15")
  },
  preferences: {
    theme: "dark",
    notifications: true,
    currency: "USD"
  },
  orderCount: 42,
  cartItems: [
    { productId: ObjectId("..."), quantity: 2, price: 29.99 }
  ]
}

Notice that cartItems is embedded—it's small, frequently updated as one unit, and read every time the user loads their cart. But orderCount is denormalized here; the actual orders live in a separate orders collection because orders grow without bound and you query them independently.

In a CRM or SaaS platform, the same principle applies. Embed contact details, account settings, and recent activity summaries. Reference historical data, audit logs, and things that scale linearly with time.

The trade-off: you'll sometimes update the same data in multiple places. That's okay. Use a background job or event stream to keep denormalized fields in sync. The consistency window of eventual consistency is vastly faster than the latency cost of joins.

Indexes Are Everything — ESR Rule, explain() and No Collection Scans

An unindexed query in production is a silent killer. It works fine on your laptop with 1,000 documents. At scale—50 million documents—it scans the entire collection and your database falls over.

We live by the ESR rule: Equality, Sort, Range. When you index fields for a query, order them this way. Equality fields first (the predicates that filter most), then sort fields, then range or regex fields. This structure lets MongoDB use the index to filter, sort in-memory, and iterate efficiently.

For a query like "find all orders for user X, created in the last 30 days, sorted by date," your index should be:

db.orders.createIndex({ userId: 1, createdAt: -1 });

userId is equality (exact match). createdAt is range (we filter on a date window) and also provides the sort order. MongoDB reads the index, finds all matching orders, sorts them with the index, and returns them. No collection scan.

Always use explain() in your queries:

db.orders.find({ userId: ObjectId("..."), createdAt: { $gte: ISODate("2024-01-01") } })
  .sort({ createdAt: -1 })
  .explain("executionStats");

Look at the output. If totalDocsExamined is much larger than nReturned, you're scanning too many documents. If stage: COLLSCAN appears, no index was used—fix that immediately.

In high-throughput systems, collection scans compound. A 10ms query that scans the collection becomes 1 second under load because the disk I/O is serialized. Your p99 latency balloons, your connection pool fills up, and cascading failures begin.

Index strategically. Don't create an index for every possible query—maintenance overhead and write latency grow. Instead, index your top 20 queries and monitor slow query logs for outliers.

Aggregation Is Powerful — Move Analytics into Pipelines, But Keep Them Indexed

The aggregation framework is where MongoDB shines. Instead of fetching data into your application and computing sums, averages, and groups in code, push the work into the database. The database is faster, uses less memory, and moves less data over the network.

A common pattern: compute daily revenue broken down by product category. In a naive approach, you'd fetch all orders for the day, loop through them in Node or Python, and accumulate sums. In MongoDB:

db.orders.aggregate([
  {
    $match: {
      createdAt: {
        $gte: ISODate("2024-01-15"),
        $lt: ISODate("2024-01-16")
      }
    }
  },
  {
    $lookup: {
      from: "products",
      localField: "productId",
      foreignField: "_id",
      as: "product"
    }
  },
  {
    $unwind: "$product"
  },
  {
    $group: {
      _id: "$product.category",
      totalRevenue: { $sum: "$total" },
      orderCount: { $sum: 1 }
    }
  },
  {
    $sort: { totalRevenue: -1 }
  }
]);

This pipeline filters, joins, groups, and sorts on the server. The application receives exactly what it needs.

But here's the catch: aggregation is only fast if your early stages can use indexes. The $match stage must run first, and it must hit an index. If it doesn't, the pipeline scans the collection and becomes a bottleneck.

A common mistake is pushing expensive operations early—like computing a derived field that requires a lookup, then filtering on it. Instead, filter first (use an index), then enrich. MongoDB will optimize the order sometimes, but you shouldn't rely on it.

Operate with Care — Replica Sets, Backups, Security and Monitoring

A well-modeled schema and fast indexes don't matter if your data disappears or leaks. Operations is where engineering meets reality.

Replica sets are non-negotiable in production. A single-node database isn't production—it's a demo. With three nodes (primary and two secondaries), you handle node failures gracefully. When the primary dies, an election happens within seconds, and a secondary takes over. Your application reconnects and continues.

Configure your replica set with a proper writeConcern. For critical data (orders, payments), use { w: "majority" }—wait until the write is replicated to a majority of nodes before acknowledging. For analytics or cache data, { w: 1 } (acknowledge on the primary only) is fine and faster.

Backups matter more than you think. A misconfigured migration script, a runaway update, or a ransomware attack can wipe your data. Use mongodump or a managed backup service. Test your restore procedure regularly—a backup that can't be restored is worthless.

Security starts with authentication. Enable authentication on day one. Use RBAC—role-based access control—so your application connects with minimal privileges. A compromised application credential shouldn't grant access to the admin database or other tenants' data.

Enable encryption in transit (TLS) and at rest (storage encryption, usually provided by your cloud). Monitor for unusual access patterns with audit logs.

Monitoring is your early warning system. Track:

• Connection count and pool utilization
• Operation latencies (read, write, command latencies at p50, p95, p99)
• Replication lag on secondaries
• Disk usage and growth rate
• Query performance—slow queries, index misses
• Memory usage on the primary

When p99 latency spikes, you need data to diagnose it. Is it a bad query, a hot shard, a GC pause, or network congestion? The answer lives in your metrics and logs.

Common Mistakes

• Designing schemas without knowing the access patterns. This is the root cause of most problems. Before you create a collection, ask: what queries will you run? How often? What latency do they need?
• Assuming indexes are free. Every index you create slows writes slightly, uses disk space, and adds maintenance complexity. Index intentionally.
• Running aggregation pipelines without a $match on an indexed field early. You'll scan the collection and wonder why it's slow.
• Skipping backups or testing restores. Data loss is rare but devastating. Backups are insurance you actually use.
• Using defaults for production deployments. Default configurations assume light load and don't prioritize durability. Adjust writeConcern, readConcern, journaling, and replica set size for your SLA.
• Not monitoring until something breaks. By then, you're blind and panicked. Instrument from the start.

Key Takeaways

• Model documents around access patterns: embed frequently-read data, reference what scales unbounded.
• Indexes are your primary performance lever. Use ESR rule, test with explain(), and eliminate collection scans.
• Push computation into aggregation pipelines, but ensure early stages use indexes for fast filtering.
• Replica sets, backups, and security aren't optional in production—they're foundational.
• Monitor latency, replication, and query performance continuously. You can't optimize what you can't see.
• The best schema is one your team understands and can maintain. Simplicity compounds.