Cluster health explained: a hands-on guide for engineers

Understanding cluster health in Elasticsearch

Your production search cluster just went red at 3 AM, and you're staring at a wall of unfamiliar API responses. Cluster health explained: a hands-on guide for engineers breaks down exactly what those status colours mean, why they matter, and how to fix them before your team notices. In this guide, we'll walk through green, yellow, and red states, then show you the diagnostic tools that turn confusion into action.

Elasticsearch cluster health is your first line of defence against silent data loss. It tells you whether your data is safe, accessible, and replicated. Most teams check it reactively—after something breaks. We're going to change that by giving you the mental model to understand what's happening and the tools to investigate any state your cluster enters.

Green — All primary and replica shards allocated

Green means your cluster is operating at full strength. Every primary shard has been assigned to a node, and every replica shard is sitting where it should be. You can lose a node and keep serving traffic. You can perform a rolling restart without downtime. This is the state you want to live in.

Here's what a green cluster response looks like:

{
  "cluster_name": "production-search",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 5,
  "number_of_data_nodes": 4,
  "active_primary_shards": 120,
  "active_shards": 360,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_check_time_ms": 45,
  "active_shards_percent_as_number": 100.0
}

Notice unassigned_shards: 0 and active_shards_percent_as_number: 100.0. That's green. If you have 3 indices with 5 primary shards each and a replication factor of 1, you'll see 15 primary shards and 15 replica shards—30 active shards total, spread across your nodes.

In a real scenario: you're running a product search index with 10 primary shards, replication factor 2 (so 30 shards total). Your cluster has 5 data nodes. During normal operation, Elasticsearch distributes those 30 shards across the 5 nodes. If one node dies, the remaining 4 nodes still hold copies of every shard. The cluster stays green because all primaries and replicas are allocated.

Green doesn't mean perfect—it means resilient. Monitor for other issues (query latency, JVM pressure, disk space) even in green. But if your cluster is green, you can ship code with confidence.

Yellow — Primaries fine, some replicas missing

Yellow means all primary shards are assigned, so your data is readable and writable. However, some replica shards haven't found a home. You're not at full redundancy. If another node dies, you might lose data.

Here's a yellow state:

{
  "cluster_name": "production-search",
  "status": "yellow",
  "timed_out": false,
  "number_of_nodes": 4,
  "number_of_data_nodes": 4,
  "active_primary_shards": 120,
  "active_shards": 240,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 120,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_check_time_ms": 32,
  "active_shards_percent_as_number": 66.7
}

See unassigned_shards: 120? Those are replica copies sitting in the queue with nowhere to go. All 120 primary shards are active, but only 240 of the 360 replica shards are allocated. In this case, we've probably lost a node (we started with 5, now have 4).

Common scenario: your 5-node cluster loses one data node (maybe a hardware failure or network partition). The primaries redistribute across the remaining 4 nodes. But there aren't enough nodes to place all the replicas. The cluster goes yellow. Your users can still read and write. But if another node fails before that missing replica is allocated, you lose data.

Yellow is a warning. It's not an emergency—you can keep the cluster running—but it's a reminder to investigate. Did you lose a node? Is it coming back? Do you need to add capacity? Is your replication factor too high for your node count?

You typically see yellow during:

• Node failure or restart
• Network partition (split-brain scenarios, though Elasticsearch prevents these with quorum rules)
• Scaling down without adjusting replication
• New index creation before all replicas initialize

Red — Some primary shards missing, data unavailable

Red is the emergency state. One or more primary shards are unassigned. That data is offline. Your cluster will reject write requests to those shards, and read requests will fail. This is data loss territory.

Here's what red looks like:

{
  "cluster_name": "production-search",
  "status": "red",
  "timed_out": false,
  "number_of_nodes": 3,
  "number_of_data_nodes": 3,
  "active_primary_shards": 110,
  "active_shards": 220,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 30,
  "delayed_unassigned_shards": 30,
  "number_of_pending_tasks": 1,
  "number_of_in_flight_fetch": 0,
  "task_check_time_ms": 28,
  "active_shards_percent_as_number": 88.0
}

The delayed_unassigned_shards: 30 tells you that 30 primary shards are waiting to be assigned. They're not just replicas—they're primaries. The data behind those shards is unreachable.

Real scenario: your 5-node cluster loses two nodes simultaneously (a failed rack or a misconfigured script). Now you have only 3 nodes left. The replication strategy can't recover. Two of your 10 indices are stored entirely on the two dead nodes. Their primary shards are unassigned. Red.

When you hit red, you need to act fast:

1. Check if lost nodes are coming back (check logs, infrastructure monitoring)
2. If they're gone for good, you may need to manually allocate those shards (knowing you might lose recent writes)
3. Rebuild from snapshots if available
4. Scale up to recover redundancy

Red states rarely happen in well-designed clusters. They happen when:

• Multiple nodes fail simultaneously
• Replication factor equals or exceeds node count (mistakes in configuration)
• All copies of a shard are on nodes that went down at the same time
• Manual shard allocation errors

Investigate — _cluster/allocation/explain shows why

When your cluster turns yellow or red, the HTTP status tells you what happened, but not why. That's where the allocation explain API comes in. It shows the decision tree for every unassigned shard.

Use this endpoint to dig into yellow or red states:

GET /_cluster/allocation/explain

{
  "index": "products",
  "shard": 0,
  "primary": true
}

The response will tell you exactly why that shard can't be allocated. Here's a typical response for a yellow cluster:

{
  "index": "products",
  "shard": 0,
  "primary": true,
  "assigned": false,
  "unassigned_info": {
    "reason": "NODE_LEFT",
    "at": "2025-01-20T14:32:10.123Z",
    "delayed": false,
    "details": "node that held the shard left the cluster"
  },
  "can_allocate": "no",
  "allocate_explanation": "cannot allocate because allocation is not permitted",
  "node_allocation_attempts": 3,
  "can_remain_on_current_node": "no",
  "worst_node_reason": "node [xyz] may not hold more than one copy of this shard",
  "manually_configured": false
}

This tells you the node left the cluster, Elasticsearch tried 3 times to find a home, but every remaining node already holds a copy of this shard. With your current node count and replication strategy, there's nowhere to put the replica.

Run allocation/explain without a specific shard to get a summary of all unassigned shards. This is your triage tool. It tells you whether the problem is capacity, node loss, or misconfiguration.

In a real incident: you lost a node, cluster went yellow, you run /_cluster/allocation/explain and learn that 50 replica shards can't be allocated because you only have 2 data nodes left and each shard already exists on both. Decision: scale up to 3 nodes, and the cluster auto-heals to green.

Common mistakes

Ignoring yellow. Teams often think "the cluster is still working, so yellow is fine." Yellow means you're one more failure away from red. Don't ignore it.

Replication factor equals node count. If you have 5 nodes and set number_of_replicas: 5, Elasticsearch can't allocate all replicas (each node would need to hold multiple copies). Keep replicas ≤ nodes − 1.

Not checking disk space. A cluster can turn yellow because a node is full, not because it's dead. The allocation explain API will tell you this, but many teams jump to "we need more nodes" without checking disk utilization.

Manual allocation without understanding shard data. When you manually force allocate a shard to recover from red, you might be allocating a stale copy. That's data loss. Always check snapshot dates and shard sync status first.

Checking health infrequently. Health should be part of your observability stack. Set an alert on anything that's not green. A weekly manual check is too late.

Key takeaways

• Green means all primary and replica shards are allocated. You're resilient to node failure.
• Yellow means all primaries are assigned, but some replicas are missing. Your data is safe, but redundancy is incomplete.
• Red means one or more primary shards are unassigned. Data is offline. This requires immediate action.
• Use GET /_cluster/health to get the overall status and shard counts.
• Use GET /_cluster/allocation/explain to understand why specific shards aren't allocated.
• Keep replica count ≤ number of nodes − 1 to allow full redundancy.
• Alert on health status changes. Yellow and red both warrant investigation.
• Disk space, node count, and replication strategy must be aligned. One problem cascades into cluster health issues.