Monitoring Elasticsearch: a hands-on guide for engineers

Elasticsearch clusters can fail silently. A node runs out of disk space, queries start timing out, or indexing throughput drops to a crawl—and you don't notice until alerts flood your on-call channel at 3 AM. Monitoring Elasticsearch: a hands-on guide for engineers means building visibility into the systems that matter most: cluster health, search performance, indexing capacity, and resource utilization. Without proactive monitoring, you're flying blind.

Key metrics — Cluster health, heap, CPU and disk

The foundation of Elasticsearch monitoring rests on understanding how your cluster is breathing. Cluster health tells you whether shards are allocated, indices are writable, and nodes are communicating. Heap memory, CPU, and disk space determine whether your cluster can sustain production load or will start rejecting requests.

Start with the Cluster Health API. It returns three states: green (all primaries and replicas assigned), yellow (primaries assigned but some replicas missing), and red (some primaries unassigned, data loss risk). Yellow is a warning sign that replica allocation is failing—often due to disk space or node count changes. Red is a critical emergency.

GET /_cluster/health

{
  "cluster_name": "production",
  "status": "yellow",
  "timed_out": false,
  "number_of_nodes": 3,
  "number_of_data_nodes": 2,
  "active_primary_shards": 150,
  "active_shards": 225,
  "relocating_shards": 10,
  "initializing_shards": 5,
  "unassigned_shards": 15,
  "delayed_uninitialized_shards": 0,
  "number_of_pending_tasks": 2,
  "number_of_in_flight_fetch": 0,
  "task_max_wait_time_in_queue_millis": 500,
  "active_shards_percent_as_number": 93.75
}

That yellow status with 15 unassigned shards indicates shard allocation is stalled. Investigate the Cluster Allocation Explain API to see why replicas aren't being placed.

Heap usage is your second critical metric. When a node's heap approaches 75%, garbage collection pauses grow longer and the cluster slows. At 95%, nodes start dropping requests. Monitor both current heap and heap max to catch trends early. A sustained climb toward 80% signals you need to add nodes or reduce indexing load.

Disk space is often overlooked until it causes a crisis. Elasticsearch enforces a watermark: by default, nodes stop accepting new shards when disk usage exceeds 85%, and start relocating shards away at 90%. If a node hits 95%, it becomes read-only. Track disk usage per node, not just cluster-wide. A single full node can cascade failures across the entire cluster.

CPU usage matters less than heap and disk, but sustained CPU above 80% indicates expensive queries, large aggregations, or under-resourced nodes. In a well-tuned cluster, CPU should hover 30–60% under normal load.

Search latency — Query and fetch times

Slow searches frustrate users and waste infrastructure. Break down search latency into two components: query time (how long the coordinator node takes to route and collect results from shards) and fetch time (how long to retrieve the actual document content).

The Nodes Stats API gives you aggregated metrics, but for per-request visibility, enable slow log queries. Set a threshold—say 1 second for your application—and log anything slower.

PUT /my-index/_settings
{
  "index.search.slowlog.threshold.query.warn": "1s",
  "index.search.slowlog.threshold.query.info": "500ms",
  "index.search.slowlog.threshold.fetch.warn": "500ms",
  "index.search.slowlog.threshold.fetch.info": "100ms"
}

Logs appear in elasticsearch-slowlog.log and include the query DSL, source, took (milliseconds), and shard count. This is invaluable for spotting pathological queries before they tank production.

Query latency spikes often come from:

• Full table scans: Queries that match millions of documents but only return 10. Use filters with must clauses instead of should for optional criteria.
• Large aggregations: Computing percentiles, terms facets, or date histograms on high-cardinality fields without constraining the result set first.
• Shard proliferation: Too many small shards mean the coordinator has to wait for many shards to respond; too few large shards mean individual shard searches take longer. Target 20–50 GB per shard for most workloads.
• Hot nodes: If one node consistently slower than others, it may be undersized, overloaded, or have a noisy neighbor process stealing CPU.

Use the Profile API on suspect queries to see exactly where time is spent:

GET /my-index/_search
{
  "profile": true,
  "query": {
    "bool": {
      "must": [
        { "term": { "status": "published" } }
      ],
      "filter": [
        { "range": { "timestamp": { "gte": "2025-01-01" } } }
      ]
    }
  }
}

The response breaks down time per shard and per query component. If a specific shard is slow, check its segment count (too many small segments slow reads) and consider a force merge.

Indexing rate — Throughput and rejected requests

Indexing is the lifeblood of search. Track both the volume of documents being indexed per second and the rate of rejected requests. A sudden drop in indexing rate usually means the cluster is overwhelmed and applying backpressure.

Monitor these metrics from the Nodes Stats API:

• indices.indexing.index_total — Cumulative number of indexing operations.
• indices.indexing.index_time_in_millis — Total time spent indexing.
• thread_pool.bulk.rejected — Count of rejected bulk requests.
• thread_pool.write.rejected — Count of rejected write operations.

Calculate indexing rate as the derivative of index_total over a time window. If your application typically indexes 10,000 docs/sec but rate drops to 2,000 docs/sec, that's a red flag—rejected requests are probably being logged by your indexing client.

Rejected requests indicate the bulk queue is full. This happens when:

• Heap pressure forces GC pauses and stalls the indexing thread pool.
• Disk I/O is bottlenecked (especially on spinning drives).
• Too many large bulk requests are queued without backoff.
• Refresh rate is too aggressive (default 1 second), causing heavy merge pressure.

Implement client-side backoff: when you see 429 Too Many Requests responses, back off exponentially. Elasticsearch itself will reject requests if the bulk queue reaches capacity, so respect that signal.

For a high-throughput pipeline, monitor indexing latency percentiles (p50, p95, p99) to catch tail latency increases before they cause queue buildup. A client library like the Elasticsearch Python client can track this natively.

Tools — Stack Monitoring or Prometheus exporters

Elasticsearch 7.0+ includes Stack Monitoring, a built-in solution for collecting and visualizing metrics. It's integrated into Kibana and requires minimal setup: enable monitoring on your cluster and optional collection agents appear automatically.

To enable Stack Monitoring:

PUT /_cluster/settings
{
  "persistent": {
    "xpack.monitoring.collection.enabled": true
  }
}

Kibana will start collecting metrics within seconds. Navigate to Stack Management → Monitoring to see node health, index performance, and shard allocation. The dashboards are purpose-built for operators and include alerts for common issues.

For multi-cluster setups or if you prefer Prometheus and Grafana, use the Prometheus Elasticsearch Exporter. It scrapes Elasticsearch APIs and exposes metrics in Prometheus format. Deploy it as a sidecar or standalone service:

docker run -d \
  -p 9114:9114 \
  -e ES_URI=http://elasticsearch:9200 \
  prometheuscommunity/elasticsearch-exporter

Configure Prometheus to scrape the exporter, then build dashboards in Grafana. Key metrics to graph:

• elasticsearch_cluster_health_status — Green/yellow/red as numeric values.
• elasticsearch_jvm_memory_used_percent — Per-node heap pressure.
• elasticsearch_filesystem_data_used_percent — Disk utilization per node.
• elasticsearch_indices_indexing_index_total — Indexing throughput.
• elasticsearch_search_query_time_milliseconds — Query latency (p99).
• elasticsearch_thread_pool_bulk_rejected — Rejected bulk requests.

Set up alerting rules for critical conditions: cluster status red, heap above 85%, disk above 90%, or rejected request rate increasing. Page on-call immediately for red status or sustained high rejection rate; warn on yellow status or trending metrics.

Common mistakes

Ignoring yellow status: Many teams treat yellow as acceptable. Yellow means data loss risk if another node fails. Investigate and fix yellow immediately, not tomorrow.

Monitoring only cluster-level metrics: A cluster can be green with 30% average disk, but one node at 95% will become read-only. Always break down metrics per node.

Tuning heap too high: Setting heap to 31 GB to "use all the RAM" disables pointer compression and slows GC significantly. Stay under 31 GB, and prefer 26–28 GB for most production clusters.

Not collecting slow logs: Slow logs are your best tool for spotting bad queries before they harm production. Enable them in non-production first, but always have them on in production at a high threshold.

Alerting on individual metrics without context: A spike in rejected requests without corresponding heap or disk pressure might be a traffic spike your cluster can handle. Correlate signals before paging on-call.

Key takeaways

• Monitor cluster health, heap usage, and disk space as your primary defense. Yellow or red status requires immediate action.
• Track search latency (query + fetch time) per shard and use slow logs to identify pathological queries before they impact users.
• Watch indexing throughput and rejected request rates; spikes in rejections indicate backpressure and often signal heap or disk stress.
• Use Stack Monitoring for a built-in solution or Prometheus exporters for flexibility; build dashboards that break metrics down per node, not just cluster-wide.
• Set up alerting with context: red cluster status and sustained high rejection rate warrant immediate paging, but investigate trending metrics before they become critical.