Common Elasticsearch mistakes: best practices explained

Introduction

We've all seen Elasticsearch clusters that were running smoothly one day and crawling the next. Often the culprit isn't a sudden spike in data or query load, but architectural decisions made months earlier that seemed reasonable at the time. Understanding common Elasticsearch mistakes: best practices explained can save your team countless hours of firefighting and prevent costly outages. In this article, we'll walk through four critical errors we've encountered in production systems and show you how to avoid them.

Relying on Dynamic Mappings

Dynamic mappings are convenient during development. You push data to Elasticsearch, and the cluster automatically infers field types. It feels frictionless until it doesn't.

The problem surfaces when Elasticsearch makes the wrong type inference. A field containing mostly integers occasionally receives a string, and Elasticsearch converts it to text. Now you have a numeric field that can't be aggregated efficiently. Or worse, you send a timestamp as a string instead of using the date type, and suddenly range queries become slow and unreliable.

We worked with a SaaS platform that relied entirely on dynamic mappings for their event ingestion pipeline. After three months, their user_id field was mapped as text in some indices and keyword in others. Aggregations that should have been instant were timing out. The fix required reindexing terabytes of data.

The solution is explicit mapping definitions. Define your mappings before you ingest data, specify exact field types, and validate incoming data shapes. Here's a practical example:

PUT /events
{
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date",
        "format": "epoch_millis"
      },
      "user_id": {
        "type": "keyword"
      },
      "event_type": {
        "type": "keyword"
      },
      "duration_ms": {
        "type": "integer"
      },
      "metadata": {
        "type": "text",
        "analyzer": "standard"
      }
    }
  }
}

Set dynamic: false or dynamic: "strict" in your mapping to catch schema violations early rather than silently accepting malformed data:

PUT /events
{
  "mappings": {
    "dynamic": "strict",
    "properties": {
      "timestamp": { "type": "date" },
      "user_id": { "type": "keyword" },
      "event_type": { "type": "keyword" }
    }
  }
}

This approach forces your application to validate before sending data to Elasticsearch, improving data quality across the board.

Too Many Small Shards

Sharding is powerful, but we often see teams over-shard their indices. The reasoning is intuitive: more shards mean parallelism and faster queries. In practice, the overhead of coordinating many small shards destroys performance.

Each shard is a Lucene segment consumer. Every query must hit each shard, merge results, and return them. With a hundred small shards, you're executing your query a hundred times just to aggregate results. Memory overhead grows, garbage collection pauses lengthen, and the cluster becomes less stable.

We optimized a trading platform's cluster that had 200 indices, each with 20 shards by default—4,000 shards total on a 6-node cluster. Most indices contained less than 500 MB of data. Reducing to 2–3 shards per index cut query latencies by 60% and eliminated their chronic GC pauses.

A simple rule: aim for shard sizes between 20 GB and 50 GB in most cases. For time-series indices, this often means 1–3 shards per day, not 10–20. Use this formula to right-size:

• Estimate daily ingest volume
• Plan for 30 days of retention
• Divide total size by 30 GB per shard
• Round up to account for replicas and overhead

Monitor shard size in your cluster:

GET _cat/shards?v=true&format=json

Look for any shard under 1 GB or over 100 GB. Those are red flags. If you have indices with uneven shard distribution—some at 5 GB, others at 50 GB—use force merge to compact segments and then reindex into properly-sized shards.

Using Deep Pagination with from

Pagination with from and size is intuitive for UI developers but terrible at scale. Requesting from: 10000, size: 20 forces Elasticsearch to fetch and sort 10,020 documents, then discard the first 10,000. On a large index, this is computationally expensive and holds memory.

An e-commerce platform we worked with had a search UI that let users browse deeply into result sets. Some power users were hitting page 500+. Each query was scanning hundreds of thousands of documents and consuming seconds of processing time. Their search infrastructure was sized for page 1–5 traffic and struggled with deeper pagination.

The practical limit for from is around 10,000 due to the index.max_result_window setting (default 10,000). Beyond that, queries fail. Even at 5,000, you'll notice latency.

Two solutions exist:

1. Search After (Recommended): Use the search_after parameter with a consistent sort order. This is stateless pagination that works efficiently at any depth.
2. Keyset Pagination: Track a cursor of the last document viewed and request the next batch relative to it.

Here's how search_after works:

POST /products/_search
{
  "size": 20,
  "query": { "match_all": {} },
  "sort": [
    { "timestamp": "desc" },
    { "_id": "asc" }
  ]
}

The response includes a sort field for each hit. Use that to fetch the next page:

POST /products/_search
{
  "size": 20,
  "query": { "match_all": {} },
  "sort": [
    { "timestamp": "desc" },
    { "_id": "asc" }
  ],
  "search_after": [1699564800000, "product_123"]
}

This approach is fast regardless of how deep users paginate. Include a tie-breaker field (_id in the example) to ensure consistent ordering across requests.

Exposing Clusters Publicly

We've seen production Elasticsearch clusters with port 9200 open to the public internet—sometimes by accident after a cloud migration, sometimes intentionally. This is a security disaster waiting to happen.

Elasticsearch has no built-in authentication at the HTTP level in older versions. Even with X-Pack security enabled, default configurations often have weak credentials or misconfigurations. An exposed cluster is an open door for data exfiltration, ransomware, and denial-of-service attacks.

A ride-hailing platform discovered their staging Elasticsearch cluster was publicly accessible. In their logs, they found queries from external IPs, bulk deletion attempts, and data extraction operations. The attacker had accessed millions of ride records and customer data before detection.

Network security rules:

• Never expose port 9200 or 9300 to the internet. Run Elasticsearch behind a private VPC.
• Use security groups or network ACLs to restrict access to application servers and admin machines only.
• If remote access is necessary, route through a VPN or bastion host.
• Enable X-Pack security and enforce strong authentication and encryption.
• Use API keys for application authentication, not usernames and passwords.
• Audit all access with security logs enabled.

In Kubernetes environments, use a service mesh or ingress controller to handle TLS termination and authentication before traffic reaches Elasticsearch. In cloud environments, use private endpoints (AWS PrivateLink, Azure Private Link) for cross-region communication.

Common Mistakes Summary

Key Takeaways

• Define explicit mappings before ingesting data and enforce schema validation with dynamic: strict to catch errors early.
• Right-size shards to 20–50 GB; too many small shards cause coordination overhead and performance degradation.
• Replace deep pagination with search_after to enable efficient browsing at any depth without scanning large result sets.
• Never expose Elasticsearch to the public internet; always run behind a private network with authentication and encryption enabled.
• Monitor cluster health regularly: shard distribution, query latencies, garbage collection, and access logs.