Disk watermarks: Elasticsearch best practices explained

Introduction

Read-only indexes are one of the most common operational emergencies in production Elasticsearch clusters. Your application starts throwing errors, monitoring alerts fire, and investigation reveals that Elasticsearch has automatically switched indexes to read-only mode because the disk is too full. Disk watermarks: Elasticsearch best practices explained reveals how to prevent this scenario through proper understanding of watermark thresholds, proactive monitoring, and a sound data retention strategy.

Disk watermarks are the safety mechanisms that Elasticsearch uses to protect your cluster from running out of disk space. When you understand how they work and configure them appropriately for your environment, you can maintain a healthy cluster and avoid the cascading failures that occur when write operations suddenly stop.

Understanding Low, High, and Flood-Stage Watermarks

Elasticsearch implements three disk watermark levels, each triggering progressively more aggressive protective actions. These watermarks are configurable thresholds expressed as either absolute byte values or percentages of total disk capacity. Understanding the distinction between them is fundamental to operating Elasticsearch reliably.

The low watermark is the first warning signal. By default, it's set to 85% of disk usage. When the cluster reaches this threshold, Elasticsearch begins moving shards away from the affected nodes, attempting to redistribute data across the cluster. This is a gentler response—the cluster remains fully functional, but it starts the process of de-risking the situation.

The high watermark comes next at 90% by default. Once disk usage crosses this line, Elasticsearch becomes more aggressive. It stops assigning new shards to this node entirely and actively relocates existing shards to nodes with more free space. Your cluster is still writable, but the pressure is mounting.

The flood-stage watermark, set to 95% by default, is the red line. When any node reaches flood-stage, Elasticsearch performs an emergency action: it sets all indexes to read-only mode cluster-wide. This means no writes are possible anywhere in your cluster, not just on the full node. Your application will immediately start failing write operations.

Here's how these thresholds appear in your cluster settings:

{
  "cluster": {
    "routing": {
      "disk": {
        "watermark": {
          "low": "85%",
          "high": "90%",
          "flood_stage": "95%"
        }
      }
    }
  }
}

You can also express watermarks as absolute values, useful in heterogeneous clusters where nodes have different disk capacities:

{
  "cluster": {
    "routing": {
      "disk": {
        "watermark": {
          "low": "100gb",
          "high": "50gb",
          "flood_stage": "10gb"
        }
      }
    }
  }
}

In this example, nodes with less than 10GB free are at flood-stage. The absolute value approach is often clearer for teams managing clusters with uniform hardware.

Why Flood-Stage Blocks All Writes

The flood-stage watermark behavior surprises many operators on first encounter. Why does Elasticsearch block writes everywhere when only one node is full? The answer lies in cluster resilience and data durability.

When you index a document, Elasticsearch must write it to a primary shard and then replicate it to replica shards across different nodes. If the cluster is running out of disk space, Elasticsearch cannot guarantee that it can durably store new data or maintain replicas. Rather than accept writes that might be lost or cause cascading failures, it takes the defensive stance of blocking all writes.

Consider a real scenario: you're indexing logs at high volume, and a single node fills up unexpectedly due to a misconfigured retention policy. Without the flood-stage protection, Elasticsearch might accept writes that it cannot replicate, leading to data loss or inconsistency. Some of your data would live on disk while copies don't exist. Flood-stage prevents this disaster by saying "we cannot continue operating safely, stop all writes immediately."

Once flood-stage is triggered, your cluster enters a crisis mode where only delete operations and shard relocation are allowed. You cannot write new data, you cannot update existing documents, you cannot even index metrics or logs.

Alert at 75% Usage Before Problems Start

Waiting for Elasticsearch to trigger watermarks is reactive. By the time the cluster hits 85% (low watermark), you're already in mitigation mode. The best-run teams alert much earlier, typically at 75% disk usage, giving themselves a full 20 percentage points of runway before any automated protective actions kick in.

This early warning approach allows for planned remediation rather than emergency firefighting. When your monitoring fires an alert at 75%, you have time to:

• Delete old indexes that have aged out of your retention window
• Archive cold data to cheaper storage or S3
• Adjust index settings like segment merge policies
• Increase cluster capacity before it becomes critical
• Communicate with application teams about temporary traffic reduction

Setting up this alert is straightforward with any monitoring stack. For Prometheus-based monitoring, an alert might look like:

{
  "alert": "ElasticsearchDiskUsageHigh",
  "expr": "elasticsearch_filesystem_data_used_percent >= 75",
  "for": "5m",
  "annotations": {
    "summary": "Elasticsearch disk usage above 75% on {{ $labels.instance }}",
    "description": "Free disk space is critically low. Begin data retention review."
  }
}

The 5-minute duration prevents spurious alerts from temporary spikes. Once this fires, your team knows they have a few hours to address the situation before Elasticsearch's automated protections engage.

For clusters running Elasticsearch 8.0+, you can also enable the data tier routing policies which automatically move older indexes to cheaper disk tiers, providing automatic disk pressure relief without manual intervention.

Delete or Move Old Data

The long-term solution to disk pressure is a sound data lifecycle management strategy. Most Elasticsearch deployments hold time-series data—logs, metrics, events—where older data has less value than recent data. Implementing a retention policy prevents unbounded disk growth.

The simplest approach is index-level deletion. If you're creating daily indexes (e.g., logs-2024-01-15, logs-2024-01-16), you can delete entire indexes older than your retention window:

{
  "schedule": "0 0 * * *",
  "action": "delete_index",
  "target": "logs-*",
  "filter": {
    "pattern": "logs-YYYY.MM.dd",
    "older_than_days": 30
  }
}

This example deletes all log indexes older than 30 days. Index Lifecycle Management (ILM) automates this process. Define a policy that moves indexes through phases:

{
  "policy": "logs-lifecycle",
  "phases": {
    "hot": {
      "min_age": "0d",
      "actions": {
        "rollover": {
          "max_primary_shard_size": "50gb"
        }
      }
    },
    "warm": {
      "min_age": "3d",
      "actions": {
        "set_priority": {
          "priority": 50
        },
        "forcemerge": {
          "max_num_segments": 1
        }
      }
    },
    "cold": {
      "min_age": "15d",
      "actions": {
        "searchable_snapshot": {
          "snapshot_repository": "s3-archive"
        }
      }
    },
    "delete": {
      "min_age": "30d",
      "actions": {
        "delete": {}
      }
    }
  }
}

In this policy, new data arrives in the "hot" phase for fast ingestion, transitions to "warm" for optimization, moves to "cold" as searchable snapshots on S3 after 15 days, and is deleted after 30 days. This keeps your primary cluster lean while preserving data for compliance.

For teams with compliance requirements to retain data longer, searchable snapshots are essential. They store index data in S3 (or other repositories) but remain searchable without keeping full copies on expensive cluster storage. Queries are slightly slower, but you've reduced disk usage by 80-90% for cold data.

Another strategy is to use data tiers explicitly. Tag your nodes with role labels:

{
  "node.roles": ["data_hot"],
  "node.tier": "data_hot"
}

Then configure ILM to migrate indexes to different tiers as they age, routing hot data to fast NVMe-backed nodes and cold data to cheaper HDD-based nodes with different disk constraints.

Common Mistakes

We've seen several patterns that lead teams into disk pressure crises:

Setting watermarks too high. Some operators raise flood-stage to 98% or 99%, thinking they're squeezing more capacity out of the cluster. In reality, they're reducing the buffer Elasticsearch needs to operate safely. Keep defaults unless you have a specific reason and strong monitoring in place.

Alerting only at watermark levels. If you only alert when Elasticsearch hits 85%, you're too late. Alerting at 75% gives you planning time. Alerting at 80% gives you emergency response time. Anything at 85%+ is crisis management.

Not automating retention policies. Teams that manually delete old indexes inevitably miss one, and disk fills up unexpectedly. Index Lifecycle Management is not optional for production clusters. Set it up early.

Ignoring replica settings during growth. As your cluster grows, increasing replica counts for resilience also increases total disk usage. If you changed replication from 1 to 2 replicas but didn't account for the disk impact, you've effectively halved your available capacity.

No capacity planning. Growing disk usage should be predictable. If you see disk usage creeping up 5% per week, you know when you'll hit problems. Capacity planning gives you the lead time to add nodes or implement retention changes before emergencies occur.

Key Takeaways

• Elasticsearch implements three disk watermarks: low (85%), high (90%), and flood-stage (95%), each triggering progressively more aggressive protective actions.
• Flood-stage sets all indexes to read-only cluster-wide, blocking all write operations—a nuclear option that prevents data loss but crashes your application.
• Alert at 75% disk usage, before any Elasticsearch watermark triggers, to give yourself time for planned remediation.
• Implement Index Lifecycle Management to automate data deletion and archival, keeping your cluster disk usage within safe bounds.
• Use searchable snapshots to retain compliance data on cheaper storage without consuming cluster disk space.
• Monitor disk usage trends over time and plan capacity increases before disk pressure becomes critical.
• Test your data retention policies in a staging environment to ensure they behave as expected at scale.