Choosing a shard key: a practical MongoDB & NoSQL guide

Choosing a shard key is one of the most consequential decisions you'll make for a MongoDB cluster at scale. Get it right, and your database grows smoothly across nodes with balanced load and predictable performance. Get it wrong, and you'll face hotspots, uneven data distribution, and a painful migration down the road.

In this guide, we'll walk through the practical framework we've used across production systems handling millions of documents, showing you how to evaluate shard key candidates and avoid the pitfalls that catch most teams.

Understanding why shard key choice matters

A shard key is the field or fields MongoDB uses to distribute documents across multiple servers in a sharded cluster. Once chosen, it cannot be changed without resharding the entire collection—a heavy operation that can lock your system during a maintenance window or, with the newer resharding feature, run in the background at significant resource cost.

The shard key determines three critical outcomes: whether data is evenly distributed across shards, whether queries can be routed efficiently to specific shards rather than all shards, and whether your write pattern creates contention at a single shard. These factors compound, so a poor choice becomes exponentially painful as you scale.

Include fields used in most queries

A shard key that appears in your most frequent queries allows MongoDB to target specific shards rather than scatter-gather across all of them. This is called a targeted query, and it's dramatically faster than a broadcast query that must hit every shard and merge results.

Consider an e-commerce platform tracking orders across regions. If most queries filter by region or tenant_id, those are strong shard key candidates because they align with your access pattern.

// Query pattern 1: Most common - all orders for a region
db.orders.find({ region: "us-west", status: "pending" });

// Query pattern 2: Common - orders by user within region
db.orders.find({ region: "us-west", user_id: ObjectId("...") });

// Query pattern 3: Rare - global scan
db.orders.find({ total: { $gt: 1000 } });

Here, region should be part of the shard key because queries 1 and 2 filter by it. A shard key of { region: 1 } or { region: 1, user_id: 1 } targets specific shards. A shard key of { total: 1 } would not help, since query 3 is rare and queries 1–2 would still scatter across all shards.

The principle is simple: if a field appears in the WHERE clause of your top 5–10 query patterns, include it in the shard key. This doesn't mean it has to be the first field in a compound key, but it should be present.

Avoid low cardinality fields

Cardinality is the number of distinct values a field can hold. A low-cardinality field like status (active, inactive, archived) or country_code (around 200 values) creates a problem: each shard receives a disproportionate share of data, and you cannot create more shards than unique values.

If you shard by status, you might have 4 distinct values. MongoDB can only use those 4 values to route data, so you're capped at roughly 4 shards. Add a 5th shard, and it sits idle. Meanwhile, if the "active" status accounts for 80% of your documents, one shard will be vastly larger than the others.

// BAD: Low cardinality
db.users.createIndex({ status: 1 });
// Only 4 distinct values: "active", "inactive", "pending", "suspended"
// This creates a bottleneck.

// BETTER: Higher cardinality
db.users.createIndex({ tenant_id: 1, user_id: 1 });
// Assuming thousands of tenant_ids and millions of user_ids,
// data and writes distribute much more evenly.

Test the cardinality of your candidates before committing. Run db.collection.aggregate([{ $group: { _id: "$field_name" } }, { $count: "count" }]) to see how many distinct values exist. A good shard key candidate should have cardinality in the thousands or millions, not single digits or tens.

Hashed keys spread writes evenly

If you have a high-cardinality field but suspect it may correlate with write patterns—for example, timestamps that are always increasing, or user IDs from a batch import that are sequential—a hashed shard key can help. MongoDB applies an MD5 hash to the field value, which randomizes the distribution and prevents hotspots from sequential writes.

A common scenario: you're logging events with a timestamp. Timestamps increase monotonically, so every write goes to the current time range. Without hashing, all recent events land on one shard, and the other shards sit idle. With hashing, the same timestamp produces different hash values for different documents (if there's any variance), distributing writes across all shards.

// Sequential writes without hash: creates a hot shard
// All writes go to the shard holding the current timestamp range
db.logs.createIndex({ timestamp: 1 });

// With hashing: writes distribute evenly
db.logs.createIndex({ timestamp: "hashed" });

// The hash converts 2024-01-15T10:30:45Z to a pseudo-random number,
// then MongoDB maps that to a shard. Next document at 10:30:46Z
// hashes to a different value, likely landing on a different shard.

The trade-off: hashed keys cannot perform range queries efficiently. If you need to find all logs between two timestamps, MongoDB must hit all shards. This is acceptable if your query patterns are mostly point lookups or if you're willing to tolerate scatter-gather for range queries.

We've used hashed keys successfully for session stores, event logs, and analytics events where write throughput is the priority and range queries are infrequent or non-critical.

Compound keys balance locality and spread

Most production shard keys are compound—two or more fields—because they balance two competing goals: routing queries to fewer shards and spreading writes evenly.

A common pattern is { tenant_id: 1, user_id: 1 }. The first field, tenant_id, is typically low cardinality relative to all possible values, but high cardinality relative to your customer base (if you have 500 tenants, that's your partition boundary). The second field, user_id, is high cardinality and prevents hotspots within a tenant.

// Example: SaaS application with multiple tenants
// Compound shard key: tenant_id first, then user_id

db.user_events.createIndex({ tenant_id: 1, user_id: 1 });

// Query 1: All events for a tenant (targeted to shards holding that tenant)
db.user_events.find({ tenant_id: ObjectId("tenant_123") });

// Query 2: Events for a specific user in a tenant (even more targeted)
db.user_events.find({ tenant_id: ObjectId("tenant_123"), user_id: ObjectId("user_456") });

// Query 3: All events across all users (must scatter-gather, but rare)
db.user_events.find({ event_type: "login" });

The ordering of fields in a compound shard key matters. MongoDB uses prefix matching: if your key is { tenant_id: 1, user_id: 1 }, queries that filter by tenant_id alone can be targeted. But if you reverse it to { user_id: 1, tenant_id: 1 }, a query filtering only by user_id must scatter across all shards, because MongoDB doesn't know which shard holds that user without also knowing the tenant.

Another approach is a hash on the second or subsequent field to prevent hotspots while keeping the first field for query targeting. For instance, { tenant_id: 1, user_id: "hashed" } lets you target by tenant but distributes writes within each tenant more evenly.

Common mistakes

• Choosing the shard key after the fact. Many teams assume they can shard later, then panic when data grows. Evaluate shard key candidates before you enable sharding, and test with production-like query patterns.
• Ignoring write hotspots. A field with high cardinality looks good until you realize 90% of writes use the same value (e.g., a global "pending" status). Always check write distribution, not just cardinality.
• Optimizing for a single query pattern. Shard keys are a compromise. If you optimize for one query, others will scatter. Review your top 10 queries and aim for a key that targets most of them, accepting that some will be slower.
• Not accounting for growth. A field with 1,000 distinct values seems fine until your business doubles and you realize only 10% of shards receive data. Cardinality should grow with your data; ideally, it scales with your intended number of shards or beyond.
• Using _id as the shard key in isolation. ObjectId is globally unique and high-cardinality, but if your queries never filter by _id, it won't help you target shards. You'll scatter-gather on almost every query.

Key takeaways

• A shard key must appear in your most frequent queries to enable targeted query routing; otherwise, you scatter-gather across all shards.
• Avoid single-digit or low-tens cardinality; aim for fields with thousands or millions of distinct values to prevent data skew and shard capacity limits.
• Hashed shard keys randomize write distribution and prevent hotspots from sequential or correlated write patterns, but sacrifice range query efficiency.
• Compound shard keys (e.g., tenant_id, user_id) are the sweet spot in most systems: the first field targets queries, subsequent fields distribute writes and prevent skew.
• Test your candidate shard keys against production query and write patterns before enabling sharding; resharding later is expensive and disruptive.
• Order matters in compound keys: prefix fields are used for query routing; ensure your most common filters match the key prefix.