Shard sizing: concepts, trade-offs and production lessons

Opening: Why shard sizing matters

We've all seen clusters that started fast and slowly ground to a halt. Often the culprit isn't missing indexes or slow queries—it's shards. Shard sizing: concepts, trade-offs and production lessons is about understanding the invisible load that accumulates when you create too many shards, split them too aggressively, or let time-series data fragment without a strategy. In this article, we'll walk through the decisions that keep clusters responsive and show you how to avoid the traps that catch teams in production.

Target size: aiming for 10–50 GB per shard

The most practical target range we've seen across production deployments is 10–50 GB per shard. This isn't arbitrary. Below 10 GB, you risk overprovisioning shards and bloating your cluster with unnecessary overhead. Above 50 GB, rebalancing becomes costly, recovery takes longer, and heap pressure from segment management increases.

Let's work through a concrete example. You're running a ride-hailing platform logging trip data: 500 MB/day, 365 days per year. That's roughly 182.5 GB annually. If you create one shard per day, you'd end up with 365 shards by year-end—each holding just 500 MB. Your cluster nodes would spend more time managing shard metadata than searching data. Now reverse it: one giant shard per year. You hit 182 GB in one shard, which means:

• Rebalancing or node failure takes hours to recover.
• Queries hit a single shard at a time, serializing work.
• Segment merging runs hot and consumes heap.

The sweet spot? Create a shard strategy that lands you around 30 GB per shard. With 500 MB/day, that's roughly 60 days per shard. So you'd have a new index every 2 months, with 1–2 shards per index. When you query, you're hitting a reasonable number of shards (not hundreds), and each shard has enough data to amortize the cost of segment management.

For reference, here's how you'd estimate shard count:

// Estimate for 500 MB/day, 30 GB target per shard
{
  "daily_ingestion_mb": 500,
  "target_shard_size_gb": 30,
  "days_per_shard": 60,
  "expected_shards_in_cluster": "~6 per year of rolling data"
}

Avoid oversharding: each shard costs memory and CPU

Here's where intuition often fails. You might think: "More shards = more parallelism = faster queries." In reality, each shard carries overhead that compounds. Every shard holds:

• Inverted index data in memory (or on disk, but with filesystem cache pressure).
• Field data structures for sorting and aggregations.
• A segment file list and metadata cache.
• Replication overhead on every replica shard.

When your cluster has 1,000 shards across 10 nodes, that's 100 shards per node. Coordinating queries across those shards means 100 network hops, 100 heap allocations, and 100 merge decisions. You've turned your cluster into a distributed coordinator instead of a search engine.

We learned this the hard way at a SaaS customer with user-generated content. They indexed each user's data in a separate index with 5 shards (for 5-node redundancy thinking). With 50,000 users, they had 250,000 shards. Queries took 2–3 seconds even for simple filters. After consolidating to per-tenant indexes with dynamic shard allocation (based on data volume), query latency dropped to 200 ms.

A practical rule: if your cluster has more shards than cores × 10, you're likely oversharded. A 10-node cluster with 40 cores each (400 total) should aim for under 4,000 shards across all indices.

To audit your shards, use the Cluster API:

GET /_cluster/stats/indices

// Returns something like:
{
  "indices": {
    "count": 45,
    "shards": {
      "total": 450,
      "primaries": 225,
      "replication": 1.0
    }
  }
}

Time-based data: using rollover to keep shards sized well

For logs, metrics, events, and analytics, time-based indexing with rollover is the standard. Rollover automatically creates a new index when your current index hits a size or age threshold, letting you control shard count predictably.

Let's say you're ingesting 2 GB/day of application logs. You want shards around 20 GB, so your rollover strategy would be:

PUT /logs-app-write
{
  "aliases": {
    "logs-app": {}
  },
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1
  }
}

POST /logs-app-write/_rollover
{
  "conditions": {
    "max_primary_shard_size": "20gb"
  }
}

With this config, after 10 days of ingestion, Elasticsearch creates logs-app-000002, and the write alias points there. Your old index is now read-only. Over 30 days, you have 3 active indexes, each with 2 shards (primary + replica), totaling 6 shards for that dataset. Simple and predictable.

But rollover isn't fire-and-forget. You need to:

• Tune max_primary_shard_size based on your node heap and query patterns. Heap-constrained clusters need smaller shards.
• Monitor rollover frequency. If you're rolling over daily when you expected weekly, something changed in your ingestion rate.
• Set retention policies. Use Index Lifecycle Management (ILM) to delete old indexes and avoid cluster bloat.

Here's an ILM policy that pairs with rollover:

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0ms",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "20gb"
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Monitor: understanding _cat/shards output

You can't optimize what you don't measure. The _cat/shards API is your primary lens into shard health and sizing.

GET _cat/shards?v=true&h=index,shard,prirep,state,docs,store,ip,node

This returns a table like:

Key things to watch:

• Shard store sizes. Are they clustered around your target (e.g., 20–30 GB), or wildly spread (some 2 GB, some 50 GB)? Wild variance suggests uneven ingestion or a failed rollover strategy.
• Unassigned shards. Any state other than STARTED is a problem. Investigate why replicas aren't being assigned.
• Node imbalance. If one node has 50 GB of shards and another has 10 GB, the cluster is inefficiently distributed. Use reroute to balance.

Set up recurring monitoring—weekly at minimum for production clusters—to catch trends. A sudden spike in shard count (from failed rollover logic) or uneven sizing (from misconfigured shard allocation) are red flags that creep up if you're not watching.

Common mistakes

Creating too many replicas for "safety". Three replicas (1 primary + 3 replicas = 4 copies) doesn't make you safer; it just wastes disk and memory. Unless you have a specific reason (zone replication), keep it to 1 replica.

Setting shard count without understanding your data growth. Don't guess. Measure your ingestion rate, calculate expected shard sizes over time, and validate before deploying to production.

Ignoring shard allocation awareness. If all your shards land on 2 nodes and you lose one, you've lost half your data. Use zone awareness and allocation rules to spread shards across failure domains.

Forgetting to delete old indexes. Over 2–3 years, time-series clusters accumulate thousands of small indexes. Use ILM retention policies religiously.

Key takeaways

• Target 10–50 GB per shard; 20–30 GB is the sweet spot for most workloads.
• Each shard carries memory and CPU cost. More shards doesn't always mean faster queries.
• Use rollover for time-series data to keep shard sizing consistent and predictable.
• Monitor _cat/shards regularly to catch uneven distribution and sizing drift.
• Pair rollover with ILM retention policies to prevent cluster bloat.
• Estimate shard count early and validate assumptions with production-like load testing.