Troubleshooting slow searches: best practices explained

Opening

Slow search queries are one of the most frustrating issues to debug in production, especially when your users are already waiting. Troubleshooting slow searches: best practices explained requires a systematic approach—we can't just guess at fixes or throw more hardware at the problem without understanding where the bottleneck actually lives. In this article, we'll walk through the tools and techniques that help us identify and resolve search latency issues in Elasticsearch.

Understanding the problem

When a search request takes longer than expected, the delay could come from many places: the query itself, how data is distributed across shards, expensive aggregations, or even resource contention on the cluster. Without visibility into each stage of query execution, we're working blind. The good news is that Elasticsearch gives us powerful diagnostic APIs that expose exactly where time is being spent.

Profile API — See which query parts take time

The Profile API is your first tool when investigating slow searches. It breaks down query execution into granular timing information, showing us how long each component takes: query parsing, rewriting, searching individual shards, and collecting results. This gives us concrete data instead of hunches.

To enable profiling on a query, add profile: true to your request:

{
  "profile": true,
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "title": "elasticsearch search"
          }
        },
        {
          "range": {
            "created_at": {
              "gte": "2024-01-01"
            }
          }
        }
      ],
      "filter": [
        {
          "term": {
            "status": "published"
          }
        }
      ]
    }
  }
}

The response will include a profile object with timing data for each shard. You'll see breakdowns like query_time_in_millis and rewrite_time. Look for:

• High rewrite times: The query is expensive to analyze. Consider simplifying boolean logic or using filters instead of queries where possible.
• Uneven shard times: One shard is much slower than others, suggesting unbalanced data distribution or a hot shard.
• Specific clause slowness: A particular must, filter, or should clause dominates the time. This tells us exactly where to optimize.

In a real-world e-commerce scenario, we profiled a product search and found that a wildcard filter on SKU was taking 40% of query time. Switching to an exact term query dropped overall latency from 800ms to 120ms.

Check shards — Too many shards per query slows search

Each query hits every shard in the index (unless you're using routing). With many shards, you're coordinating requests across more nodes, merging more results, and increasing the risk of one slow shard delaying the entire request. This is often called the "coordination overhead."

The rule of thumb: aim for 1–3 shards per node in your cluster, and keep individual shard sizes between 20–50 GB. Too many small shards means too much coordination; too few large shards means poor parallelism and slow individual shard searches.

Check your current shard layout with the indices stats API:

GET /_cat/indices?v&h=index,shards,docs.count,store.size

This gives you shard count, document count, and storage size per index. If you see an index with 20 shards and only 10 million documents across a 3-node cluster, that's likely wasteful. Conversely, if a single shard is 100 GB and search is slow, you're probably not parallelizing enough.

When you have too many shards, search requests spawn many parallel tasks. Coordinating all those tasks has overhead. Merging results from 50 shards takes longer than merging from 5. We've seen teams reduce p99 latency by 30–40% just by consolidating shards.

If you inherit an over-sharded index, you have a few options:

1. Reindex into a smaller number of shards: Use the reindex API and set number_of_shards: 3 (or an appropriate number for your cluster size).
2. Use index.number_of_replicas instead: Replicas don't increase shard count; they add read capacity without coordination overhead.
3. Accept it for now, but plan a rollover: For time-series indices, the next rollover can use better shard settings.

Heavy aggregations — High-cardinality terms aggregations are costly

Aggregations can be expensive, especially when you're aggregating on high-cardinality fields (fields with many unique values, like user IDs or timestamps). Every shard must compute its portion of the aggregation, and the coordinator must then merge results from all shards.

A common mistake is writing something like this:

{
  "size": 0,
  "aggs": {
    "users": {
      "terms": {
        "field": "user_id",
        "size": 10000
      }
    }
  }
}

If user_id has millions of unique values, Elasticsearch must track all of them in memory on every shard, then merge. This is memory-intensive and slow. A few strategies:

• Lower the size: Use "size": 100 instead of 10,000 if you only need top results. Elasticsearch can optimize for smaller sets.
• Add min_doc_count: Set "min_doc_count": 10 to exclude rare values. This reduces the working set significantly.
• Use composite aggregations for pagination: Instead of fetching all buckets at once, paginate through them. This is essential for real-time dashboards.
• Pre-aggregate at index time: If you're always counting events per user per day, store that in a separate aggregated index rather than aggregating raw events on demand.

Here's a better version:

{
  "size": 0,
  "aggs": {
    "top_users": {
      "terms": {
        "field": "user_id",
        "size": 100,
        "min_doc_count": 5
      }
    }
  }
}

We worked with a SaaS team running analytics queries on billions of events. They were timing out on terms aggregations over user IDs. By switching to composite aggregations with pagination and setting sensible defaults, they transformed a 30-second query into three 1-second requests that could be paginated through the UI.

Hardware — Disk and memory pressure affect everything

Elasticsearch is a resource-hungry application. When your cluster runs low on memory or disk space, search performance tanks across the board—not just for one index, but for everything.

Memory pressure: Elasticsearch uses memory for caching, query state, and aggregation results. If you're consistently above 80% heap usage, queries slow down because the JVM spends more time on garbage collection. Use the _nodes/stats API to check heap:

GET /_nodes/stats/jvm

Look at heap_used_percent. If it's regularly over 75%, you need more nodes or larger heap. Don't just increase heap size blindly—that makes GC pauses longer. Add more nodes instead.

Disk space: When a node runs out of disk, Elasticsearch may stop allocating shards to it, putting more load on other nodes. It can also trigger the disk.watermark.flood_stage (default 95%), which makes the cluster read-only. Always leave headroom. Monitor /_cat/nodes?v&h=name,disk.percent,heap.percent regularly.

Query cache and filter cache: Elasticsearch caches filter results in memory. With many concurrent searches, cache contention is real. You can monitor cache hit rates with:

GET /_stats/indices?groups=_all

If you see low query cache hit rates and high memory usage, consider:

• Reducing indices.queries.cache.size if you have many different queries.
• Adding more nodes to distribute load.
• Using read-only index replicas in separate node roles for expensive analytical queries.

In one e-commerce deployment, search latency kept creeping up over time. Profiling showed query times were stable, but coordination overhead was increasing. The issue: disk was filling up, and Elasticsearch was redistributing shards constantly. Once we added storage and rebalanced, latency dropped by half.

Common mistakes

Not using the Profile API first: We sometimes see teams tweaking settings or adding hardware without ever running a profile. This is guesswork. Always profile before you optimize.

Ignoring shard imbalance: A cluster where one shard has 100 GB and another has 10 GB will perform as badly as the slowest shard. Rebalancing can be painful but is worth it.

Aggregating high-cardinality fields without limits: Asking for the top 100,000 users by document count will consume gigabytes of memory. Always set reasonable bounds.

Assuming more shards = faster: This is the opposite of the truth. More shards add coordination overhead. The sweet spot is usually 1–3 per node.

Overlooking system resources: A cluster at 90% heap usage will be slow no matter how well-tuned your queries are. Monitor infrastructure metrics alongside Elasticsearch metrics.

Key takeaways

• Always start with the Profile API to see exactly where time is spent in your query.
• Keep shard count proportional to your cluster size—typically 1–3 shards per node.
• Limit high-cardinality aggregations with size and min_doc_count to reduce memory and CPU.
• Monitor heap usage, disk usage, and cache hit rates continuously; slow searches often signal resource pressure.
• Rebalance shards if sizes are uneven; a 10x difference in shard size means a 10x difference in query latency.
• Use composite aggregations for pagination instead of fetching all buckets at once.
• Pre-aggregate common queries at index time if they're run repeatedly.