Elasticsearch cost optimization: lessons from production

Introduction

Most teams reach a moment where their Elasticsearch bill becomes uncomfortably large. Whether you're running a high-volume e-commerce search index, a logging platform, or a real-time analytics cluster, costs can spiral quickly if you're not deliberate about data retention, shard allocation, and storage tiers. This guide covers Elasticsearch cost optimization: lessons from production—practical strategies we've used to cut monthly cloud spend by 40–60% without degrading search performance or losing data.

The good news: cost optimization is not a one-time project. It's a set of repeatable patterns you can apply during cluster design, ongoing operations, and when reviewing old data.

Delete or tier old data

The simplest way to reduce costs is to stop paying for data you don't need. In production, we rarely query logs or events from more than a few weeks ago, and business records older than a few months move into compliance storage rather than active indices.

Index Lifecycle Management (ILM) automates this. You define policies that move indices through phases: hot (ingest-heavy, replicated), warm (read-only, fewer shards), cold (rarely accessed, highly compressed), and delete (purged after retention window).

Here's a real example: a SaaS platform ingesting 50 GB per day of application logs. Without ILM, they kept 90 days of logs in a single cluster at full replication. By setting a 30-day hot phase, 30-day warm phase, and automatic deletion after 60 days, they halved storage costs immediately. The 90-day logs still existed in S3 for compliance, but Elasticsearch only held what the team actually queried.

{
  "policy": "logs-lifecycle",
  "phases": {
    "hot": {
      "min_age": "0d",
      "actions": {
        "rollover": {
          "max_primary_size": "50gb",
          "max_age": "1d"
        },
        "set_priority": {
          "priority": 100
        }
      }
    },
    "warm": {
      "min_age": "7d",
      "actions": {
        "set_priority": {
          "priority": 50
        },
        "forcemerge": {
          "max_num_segments": 1
        }
      }
    },
    "cold": {
      "min_age": "30d",
      "actions": {
        "set_priority": {
          "priority": 0
        },
        "searchable_snapshot": {}
      }
    },
    "delete": {
      "min_age": "60d",
      "actions": {
        "delete": {}
      }
    }
  }
}

Use DELETE by time window, not by query volume. Query-based deletion (DELETE_BY_QUERY) is CPU-intensive and fragments indices. Instead, roll indices daily or hourly, and delete entire index shards when their retention window closes.

Right-size shards and nodes

Undersized or oversized shards drive unnecessary costs. Too many small shards consume heap memory for metadata; too few large shards make rebalancing slow and shard recovery expensive. Each shard typically needs 1–5 GB of heap, and each node in a 3-node cluster costs the same whether it holds 10 shards or 100.

The practical rule: aim for shards between 10 GB and 50 GB, depending on query patterns. For write-heavy indices (logs, events), 20–30 GB per shard is common. For read-heavy indices (product catalog, user profiles), 40–50 GB works well.

We once audited a trading platform with 500-shard cluster holding 2 TB of data—averaging 4 MB per shard. When nodes restarted, recovery took hours because so many tiny shards needed to reassign. By merging down to 50 shards (20 GB each), recovery dropped to 20 minutes, and node memory pressure fell by 30%.

Calculate your target shard count:

1. Estimate total data size: 2 TB
2. Choose shard size: 30 GB
3. Divide: 2,000 / 30 ≈ 67 shards
4. Multiply by replicas + 1: if 1 replica, then 67 × 2 = 134 total shard instances
5. Spread across nodes: 134 / 3 nodes ≈ 45 shards per node (acceptable)

Use the Cluster Allocation Explain API to spot imbalance:

GET _cluster/allocation/explain
{
  "index": "products",
  "shard": 0,
  "primary": false
}

Node sizing follows similar logic. A 3-node cluster with 32 GB heap per node is more cost-effective than a 6-node cluster with 16 GB each—you get the same total resources with fewer coordination overhead. However, go below 3 nodes and you lose resilience; go above a dozen per tier and you're paying for nodes that sit idle during off-peak hours.

Use frozen tiers for archives

Searchable snapshots (Elasticsearch 7.12+) and the frozen tier (7.15+) let you archive old indices at a fraction of the cost. Data is stored on object storage (S3, GCS) and loaded into cache on demand. It's slower—queries take 10–100 ms longer—but acceptable for compliance data or infrequent reports.

A travel company stored 5 years of booking history in frozen tier. Queries ran at 90th percentile in ~500 ms instead of 50 ms on warm tier, but the cluster used 10× less memory and storage was 80% cheaper. For their use case—monthly reporting and occasional audits—the trade-off was worthwhile.

To move an index to frozen:

PUT /bookings-2019-01/_settings
{
  "index.tier.prefer": "frozen"
}

Frozen tier requires dedicated nodes with good network bandwidth to object storage. Allocate a small node pool (2–3 nodes) with 64 GB heap; they'll cache frequently accessed shards in memory and spill to disk as needed.

One caveat: you cannot index into frozen shards. Use it only for time-series data that's never updated after a retention window.

Avoid storing unused fields

Every field in every document consumes memory, disk, and inversion index overhead. Many teams ingest extra fields "just in case" they'll need them later. In practice, 20–30% of indexed fields are never queried.

Do a field usage audit. Enable slow query logs and query profiling:

PUT _cluster/settings
{
  "transient": {
    "search.max_buckets": 65535,
    "logger.deprecation": "warn"
  }
}

GET /my-index/_search?explain=true
{
  "size": 0,
  "query": {
    "match_all": {}
  }
}

Then analyze which fields appear in search requests, aggregations, and sorting over a week. Fields used in fewer than 1% of queries are candidates for removal.

A ride-hailing platform was storing raw GPS coordinates, device IDs, and raw event JSON strings in every ride index. When they removed low-cardinality fields and stored raw payloads in a separate cold archive, index size dropped 35% and indexing throughput improved 20%. Queries weren't affected because they never filtered on those fields.

Use _source filtering to exclude fields from storage:

PUT /rides
{
  "mappings": {
    "_source": {
      "excludes": ["raw_payload", "debug_info"]
    },
    "properties": {
      "ride_id": { "type": "keyword" },
      "fare": { "type": "scaled_float", "scaling_factor": 100 },
      "pickup": { "type": "geo_point" },
      "dropoff": { "type": "geo_point" }
    }
  }
}

Be cautious: excluded fields cannot be retrieved in results or used for filtering. If you need that data occasionally, store it in a separate document or archive rather than excluding it.

Common mistakes

• Over-replication. Three replicas might feel safe, but for time-series data with a 7-day retention window, one replica is usually enough. You're not protecting against permanent loss; you're just adding cost.
• Keeping all historical indices in hot tier. Old indices should roll to warm or cold automatically. Manually managing ILM policies for hundreds of indices is error-prone.
• Ignoring shard metadata overhead. A 100-shard cluster with 1 MB per shard totals 100 MB of metadata that lives in heap on every node. Consolidate shards regularly.
• Using high refresh rates on non-critical indices. Logs don't need refresh_interval: 1s. Set it to 30 or 60 seconds for bulk indices; 1 second is only necessary for real-time monitoring.
• Storing formatted text instead of numeric codes. If you're indexing country names as text, you're wasting space. Use a keyword field with a small numeric ID and decode on the client.

Key takeaways

• Use Index Lifecycle Management to automatically delete or tier old data. Retention policies are the single biggest cost driver.
• Target 20–50 GB per shard and 3+ nodes per cluster. Right-sizing shards reduces heap overhead and improves recovery time.
• Archive old indices to frozen tier or searchable snapshots for compliance data. Slower queries are acceptable when you query rarely.
• Audit field usage monthly. Remove fields that never appear in filters, aggregations, or sorts.
• Reduce replica count for time-series data with short retention. You're not buying durability; you're buying availability.
• Set appropriate refresh intervals. Most indices don't need sub-second freshness.
• Monitor node heap usage and shard allocation regularly. Imbalance creeps in gradually and costs accumulate silently.