Replicas and availability: practical tips and trade-offs

Introduction

Node failures in production are not a matter of if, but when. Whether it's a hardware fault, network partition, or planned maintenance, your Elasticsearch cluster will eventually lose nodes—and without proper replica configuration, you'll lose data access alongside them. Understanding replicas and availability: practical tips and trade-offs is essential for building resilient search infrastructure that survives real-world disruptions.

Over the past 37 days, we've built indexing strategies, query optimization, and monitoring foundations. Today, we lock in the operational practices that keep your cluster serving traffic when nodes go down. By the end of this article, you'll know exactly how many replicas to run, where to place them, and how to interpret cluster health signals that tell you something is wrong.

At least one replica in production

The golden rule of production Elasticsearch is simple: every shard must have at least one replica. A primary shard with zero replicas is a single point of failure. If that node crashes, your data for that shard becomes inaccessible until the node recovers or you manually intervene.

Consider a typical mid-sized e-commerce search index. Your product catalog index has 5 primary shards. Without replicas, losing one node takes 20% of your search capacity offline instantly. With one replica per shard, you have 10 total shards (5 primary + 5 replica), spread across at least two nodes. Now a single node failure means you lose 10% of your shards, but queries still route to the surviving replica shards on other nodes.

The index configuration is straightforward:

PUT /products
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 1
  }
}

This creates 5 primary shards and 1 replica per primary, totaling 10 shards across your cluster. Elasticsearch will automatically distribute these replicas across different nodes. If you have exactly 2 nodes, you'll have roughly 5 shards on each node—some primary, some replica.

Most teams we've worked with in production run at least one replica for every index receiving significant query traffic. Logs and time-series data that are indexed once and queried heavily benefit enormously from this baseline. The disk cost is roughly 2x your raw data size, but the availability gain justifies it for any mission-critical search tier.

Replicas serve reads

A common misconception is that replicas exist only for failover. In reality, replicas actively participate in read operations. When you execute a search query, Elasticsearch picks one shard copy—primary or replica—and routes the request to it. This distributes query load across all copies of a shard, not just the primary.

This has real implications for capacity planning. With 5 primary shards and 1 replica each, you have 10 shard copies. Search queries can be served by any of those 10 copies. If your query throughput is uniform, you're effectively doubling the query capacity compared to zero-replica setup.

Let's say your ride-hailing service runs a user-search index (finding drivers or passengers) with the following setup:

PUT /users
{
  "settings": {
    "number_of_shards": 3,
    "number_of_replicas": 1,
    "refresh_interval": "1s"
  }
}

You have 6 total shard copies. During peak hours, incoming queries are routed to any available shard copy. The routing algorithm in Elasticsearch attempts to balance the distribution, sending roughly 1/6 of traffic to each shard copy. This means each physical shard handles ~50% less query load than it would with zero replicas.

However, writes only go to the primary shard. Replicas receive write operations after the primary has acknowledged them. This creates a small lag between primary and replica updates—typically milliseconds—which is acceptable for most search workloads but important to understand for real-time consistency requirements.

Spread replicas across zones

Replicas are only valuable if they survive the same failures as your primary shards. If your primary and replica shards live on the same physical machine or the same availability zone, a zone-wide outage takes both offline simultaneously. You've added disk overhead without gaining resilience.

Production clusters must use shard allocation awareness to ensure replicas spread across failure domains. In AWS, this typically means different availability zones. In on-premise setups, this might mean different racks or physical locations.

Configure awareness in your elasticsearch.yml:

cluster.routing.allocation.awareness.attributes: zone
node.attr.zone: us-east-1a

Then in your index settings, enforce that different zone replicas are required:

PUT /products
{
  "settings": {
    "number_of_shards": 5,
    "number_of_replicas": 2,
    "cluster.routing.allocation.awareness.force.zone.values": "us-east-1a,us-east-1b,us-east-1c"
  }
}

With this configuration and three availability zones available, Elasticsearch will place primary shards in one zone, replica copies in different zones. If us-east-1b goes down entirely, all data remains available from other zones.

We've seen teams run a two-zone setup with one replica and survive zone failures consistently. The trade-off is that you need at least two zones, and your cluster continues degraded when one zone is unavailable (all reads serve from the surviving zone, which increases latency and CPU). For truly critical systems—travel booking, payment processing—three zones with two replicas ensures a single zone failure barely impacts availability.

Yellow health means missing replicas

Elasticsearch reports cluster health in three states: green, yellow, and red. Understanding what each means prevents silent availability degradation.

• Green: All primary shards and all replica shards are allocated. Your cluster is at full capacity.
• Yellow: All primary shards are allocated, but some replicas are missing. You're still serving all queries, but you've lost fault tolerance.
• Red: Some primary shards are missing. You cannot serve queries for those shards; they're inaccessible.

Check your cluster health with:

GET /_cluster/health

A typical response when everything is fine:

{
  "cluster_name": "production-search",
  "status": "green",
  "timed_out": false,
  "number_of_nodes": 5,
  "number_of_data_nodes": 5,
  "active_primary_shards": 15,
  "active_shards": 30,
  "relocating_shards": 0,
  "initializing_shards": 0,
  "unassigned_shards": 0,
  "delayed_unassigned_shards": 0,
  "number_of_pending_tasks": 0,
  "number_of_in_flight_fetch": 0,
  "task_max_waiting_in_queue_ms": 0,
  "active_shards_percent_as_number": 100
}

Yellow health often appears during node maintenance or after a node restart. For example, you update a data node. It rejoins the cluster, shards start rebalancing, and temporarily some replicas haven't reassigned yet—status turns yellow. This is usually temporary and resolves within minutes.

However, if your cluster stays yellow for hours or days, investigate. Common causes include:

• Not enough nodes to place all replicas (you set 2 replicas but only have 2 nodes total)
• Disk space warnings preventing shard allocation
• Misconfigured awareness attributes
• Unassigned shards stuck waiting for specific node attributes that don't exist

We typically alert on yellow health persisting longer than 15 minutes. Most transient issues clear faster. Prolonged yellow indicates configuration issues that need investigation before they become red.

Common mistakes

Setting replicas too high without planning capacity: Teams sometimes configure 3 or 4 replicas per shard thinking "more resilience is always better." With 4 replicas per primary, you need 5x the disk space. If your cluster was already tight on disk, this triggers cascading allocation failures and ironically makes availability worse. Start with 1 replica, scale to 2 only if your business criticality and node count justify it.

Ignoring zone distribution: Configuring replicas without awareness attributes gives you no real protection. A full-zone failure still takes your primary and replicas offline together. Always configure awareness when you have multiple failure domains available.

Reacting to yellow instead of preventing it: Many teams see yellow health and immediately add more nodes or increase replica counts reactively. A better approach is monitoring your node count versus replica configuration ahead of time. If you run 3 nodes with 2 replicas, you'll hit yellow when you lose any node. Plan your topology intentionally.

Forgetting about resharding: When you add nodes to a cluster, existing shards rebalance to use the new capacity. During this rebalancing, network and CPU spike. Some teams schedule this during off-peak windows. Others add nodes during off-hours to avoid impacting production queries. Don't just assume rebalancing is instant.

Key takeaways

• One replica per shard is non-negotiable for production; zero replicas means one node failure takes data offline.
• Replicas serve reads, not just failover; they increase query capacity proportionally to the number of replicas.
• Use shard allocation awareness to spread replicas across availability zones or failure domains; otherwise replicas don't protect against zone outages.
• Yellow cluster health means missing replicas; investigate if it persists longer than 15 minutes, as it indicates configuration or capacity issues.
• Plan your replica count and node count together; more replicas require proportionally more nodes to stay healthy.
• Monitor and alert on cluster health state, unassigned shards, and initializing shards as early warning signals for availability problems.