Index lifecycle management: patterns, pitfalls and fixes

Index lifecycle management: patterns, pitfalls and fixes

When you're shipping logs and time-series data to Elasticsearch at scale, you quickly realize that letting everything live in a single index forever is a path to operational chaos. Index lifecycle management (ILM) is the framework that lets us automate the journey of data from hot, queryable state through to eventual deletion, keeping cluster performance stable and costs predictable. In this guide, we'll walk through the lifecycle phases, rollover strategies, optimization techniques, and the real pitfalls we've encountered in production.

Understanding the five lifecycle phases

Elasticsearch's ILM divides an index's lifetime into up to five phases: hot, warm, cold, frozen, and delete. Each phase represents a trade-off between query performance, storage efficiency, and cost. The phases aren't mandatory—you design your policy around what your data needs.

The hot phase is where all writes happen. Your index is actively receiving new documents, and queries need to complete in milliseconds. Hardware here is typically fast SSD storage with plenty of CPU and RAM. You might keep an index hot for a few hours or a few days depending on your ingestion rate and query patterns.

The warm phase begins when an index stops receiving writes but is still regularly queried. We typically move here based on age or size. Hardware can be slightly less performant than hot—perhaps larger but fewer shards, still on SSD but in a different tier. Warm nodes might still serve dashboards and occasional ad-hoc searches.

The cold phase is for data that's rarely accessed—compliance checks, historical analysis, or fallback searches. We aggressively optimize here: reduce replica count, apply force merge, potentially shrink the shard count. Cold nodes might use larger, slower storage. Query latency is acceptable at seconds rather than milliseconds.

The frozen phase is Elasticsearch's answer to "we need to keep this for seven years but almost never touch it." Frozen indices use searchable snapshots, keeping only the data needed to answer a query in cache and fetching the rest from a blob store on demand. This is dramatically cheaper than keeping data in active cluster storage.

The delete phase simply removes the index when retention expires. For compliance or business reasons, sometimes you can't delete immediately—you might snapshot first, then delete after a delay.

Rollover: triggering index transitions

Rollover is the mechanism that creates a new index and moves the write alias to it, typically when an index reaches a size or age threshold. This prevents any single index from becoming too large or too old, which helps with query performance and operational flexibility.

You define rollover conditions in your ILM policy. A common pattern is to roll over indices daily for logs, or when they reach 50 GB, whichever comes first:

{
  "policy": "logs-policy",
  "phases": {
    "hot": {
      "min_age": "0ms",
      "actions": {
        "rollover": {
          "max_primary_shard_size": "50gb",
          "max_age": "1d"
        }
      }
    }
  }
}

In production, we've found that rolling based on primary shard size (rather than total index size) is more predictable. A primary shard size of 40–60 GB tends to be a sweet spot: large enough to minimize the number of shards, but small enough that recovery and rebalancing don't cause cascading timeouts.

One gotcha: if you're ingesting faster than you expect, rollover might happen every few hours instead of daily. Monitor your rollover frequency and adjust thresholds. Also, ensure your write alias is correctly set up before ILM takes over—mistakes here can cause writes to go to the wrong index or fail silently.

Shrink and force merge: optimizing older indices

Once an index moves out of the hot phase, we optimize it aggressively. Force merge combines Lucene segments into fewer, larger segments, reducing memory overhead and speeding up queries on immutable data. Shrink reduces the number of primary shards, which is useful when you've split an index heavily during hot ingestion but don't need that parallelism anymore.

A typical warm-phase ILM action looks like this:

{
  "warm": {
    "min_age": "3d",
    "actions": {
      "set_priority": {
        "priority": 50
      },
      "shrink": {
        "number_of_shards": 2
      },
      "force_merge": {
        "max_num_segments": 1
      }
    }
  }
}

The set_priority action ensures warm indices are deprioritized during recovery or rebalancing, so hot data doesn't get starved. Shrinking from 12 shards (for fast ingestion) down to 2 or 3 shards cuts memory footprint significantly. Force merging to 1 segment can reduce query latency by 20–40% on warm data, since there are fewer Lucene segments to search.

Shrink and force merge both require the index to not receive writes. That's why they happen after rollover, when the index is no longer the write target. Watch your cluster disk space during force merge—it temporarily increases usage as segments are rewritten. On a 500 GB warm index, force merge can temporarily consume another 500 GB while it runs.

Delete: removing data at retention boundaries

The delete phase removes indices when they're no longer needed. In many organizations, retention is driven by compliance (GDPR, HIPAA, financial regulations) or simply by business policy ("keep logs for 90 days"). ILM lets you automate this so data doesn't linger past its retention window.

A retention-driven delete policy might look like:

{
  "delete": {
    "min_age": "90d",
    "actions": {
      "delete": {}
    }
  }
}

Deleting an index is fast—Elasticsearch just removes the metadata and tells the filesystem to reclaim space. However, if you need an audit trail or must keep data offline for compliance, you can snapshot the index first, then delete it later. A two-phase approach looks like:

{
  "delete": {
    "min_age": "30d",
    "actions": {
      "snapshot": {
        "repository": "s3-backup",
        "snapshot": "logs-snapshot"
      }
    }
  }
},
{
  "delete": {
    "min_age": "91d",
    "actions": {
      "delete": {}
    }
  }
}

This snapshots at 30 days and deletes at 91 days, giving you 61 days of offline backup. Make sure your snapshot repository has adequate capacity and that you actually test restores periodically.

Common mistakes

Misconfiguring shard count: Starting with too many shards in hot indices leads to excessive memory overhead and coordination overhead. Too few shards bottlenecks ingestion. Begin with 1–2 shards per index and monitor CPU and disk I/O; scale up only if needed.

Ignoring force merge consequences: Force merging a 100 GB index to 1 segment will spike cluster CPU and disk I/O. Run it during low-traffic windows or stagger it across multiple warm indices. Monitor JVM heap during merge operations.

Forgetting about replicas in cold phase: Keeping replicas on cold nodes wastes storage. Set replica count to 0 in cold and frozen phases unless you have specific disaster recovery requirements. If a cold node dies, re-index from the warm replica or snapshot.

No monitoring of ILM progress: ILM policies can silently fail if a node label is missing, a shrink target is invalid, or a snapshot repository is unreachable. Monitor the ILM explain API and cluster state regularly. Set alerts on policy execution errors.

Mixing hot and warm nodes inappropriately: If hot and warm nodes share hardware, ILM will shuffle data between them and they'll compete for CPU and I/O. Use separate node roles (data_hot, data_warm, data_cold) and tag them in your node attributes, then reference those tags in ILM policies via allocation rules.

Key takeaways

• Index lifecycle management automates the journey of data through hot, warm, cold, frozen, and delete phases, keeping your cluster performant and costs predictable.
• Rollover creates new indices at size or age thresholds; aim for 40–60 GB primary shard size as a practical default and monitor rollover frequency.
• Shrink reduces shard count and force merge consolidates segments in warm and cold phases, lowering memory and query latency with minimal risk.
• Delete policies enforce retention windows automatically; use snapshots if compliance requires offline archives before final deletion.
• Tag your nodes by tier (hot, warm, cold) and reference those tags in ILM allocation rules to prevent resource contention.
• Monitor ILM execution and cluster state; silent failures in policy transitions are common and hard to debug without visibility.