JVM heap sizing: Elasticsearch best practices explained

Understanding Elasticsearch memory configuration

Elasticsearch clusters fail silently when memory pressure builds up, and by the time you see latency spikes, you're already in production chaos. JVM heap sizing: Elasticsearch best practices explained is one of those foundational topics that separates clusters running smoothly for months from those hitting out-of-memory errors weekly. Getting heap size wrong doesn't just cause crashes—it creates garbage collection pauses that tank query performance and destabilize your whole search infrastructure.

Memory tuning isn't about maxing out heap allocation. It's about understanding how the JVM, the operating system, and Elasticsearch's caching layers work together. We've seen teams provision 256 GB of RAM per node and still hit memory issues because they didn't account for how heap pressure affects garbage collection and file system cache efficiency.

Heap at most half of RAM

The first rule of Elasticsearch memory management is this: never allocate more than half of your node's available RAM to JVM heap. If your server has 64 GB of RAM, set heap to 32 GB maximum—not 48 GB, not 60 GB. The reason is straightforward: Elasticsearch depends heavily on the operating system's file system cache to serve hot data without disk I/O.

When heap is too large, it starves the file system cache. Elasticsearch will have plenty of memory for its data structures, but the kernel won't have room to cache index segments on disk. This forces reads to go directly to storage, which is orders of magnitude slower than reading from RAM via the page cache. You end up with a paradoxical situation: more heap feels safer but performs worse because every query now competes for the same RAM.

Consider a 128 GB node. The instinct is to give Elasticsearch 96 GB or even 100 GB of heap. Wrong. Set it to 64 GB. The remaining 64 GB becomes file system cache for your index segments. Elasticsearch writes data structures to disk, the kernel caches those segments, and queries hit RAM hundreds of times faster than if the kernel has no cache to work with.

In your elasticsearch.yml configuration:

# For a 128 GB node
-Xms64g
-Xmx64g

# For a 64 GB node
-Xms32g
-Xmx32g

# For a 32 GB node
-Xms16g
-Xmx16g

Notice we set both Xms and Xmx to the same value. This prevents heap resizing during runtime, which triggers stop-the-world GC pauses that can freeze your cluster for seconds or minutes.

Stay under 31 GB for compressed object pointers

Compressed object pointers (CompressedOops) is a JVM feature that reduces memory overhead when heap stays below a specific threshold. With CompressedOops enabled, Java object references use 32 bits instead of 64 bits, cutting memory consumption by roughly 30 percent. Once heap exceeds 31 GB, CompressedOops is disabled and pointers expand to 64 bits, wasting significant memory on the pointer infrastructure itself.

This is why the industry standard recommendation is: if you must use a single node with large heap, cap it at 30 GB rather than 32 GB. The memory savings from CompressedOops are substantial enough that a 30 GB heap with compression often outperforms a 32 GB heap without it.

Many production teams follow a two-heap strategy instead: use two or more medium-sized nodes with 15–20 GB heap each, rather than one large node with 64 GB. Each node gets the benefits of CompressedOops, the cluster gains redundancy if a node fails, and you can perform rolling updates without losing availability.

You can verify CompressedOops is active by checking the Elasticsearch logs on startup:

# Good: CompressedOops is enabled
[2024-01-15T10:22:33,456][INFO ][o.e.j.JvmInfo] heap size [29.9gb], 
compressed ordinary object pointers [true]

# Bad: CompressedOops is disabled
[2024-01-15T10:22:33,456][INFO ][o.e.j.JvmInfo] heap size [32.1gb], 
compressed ordinary object pointers [false]

If you see compressed pointers marked false and your heap is above 31 GB, consider splitting the node or reducing heap allocation. The performance gain from staying under the threshold is worth restructuring your deployment.

Leave the rest for the file system cache

The file system cache is where most Elasticsearch magic happens. When Lucene writes inverted index segments to disk, the kernel automatically caches frequently accessed pages in RAM. Queries then hit this cache instead of making expensive disk reads. The larger the cache, the higher the proportion of your working set stays in RAM.

A 128 GB node with 64 GB heap means 64 GB is reserved for the file system cache. That's typically enough to cache your most active shards and segments. A 128 GB node with 120 GB heap leaves only 8 GB for caching—nearly everything goes to disk after the first hit, destroying performance.

The relationship between heap and file system cache is not linear. As long as you have some cache, you benefit. But the benefits scale quickly at first. Going from zero cache to 16 GB cache is transformative. Going from 32 GB to 48 GB cache provides incremental improvement. Most teams find that equal split—half heap, half cache—provides the best balance for typical workloads.

Monitor cache hit ratios to validate your split is working:

GET /_nodes/stats/fs

{
  "nodes": {
    "abc123": {
      "fs": {
        "timestamp": 1705329753456,
        "total": {
          "total_in_bytes": 137438953472,
          "free_in_bytes": 68719476736,
          "available_in_bytes": 64424509440
        }
      }
    }
  }
}

If available space shrinks as your cluster serves traffic, it means the kernel is using that space for caching—exactly what you want. If available stays high, the working set isn't fitting in cache and you need either fewer shards per node or larger nodes with better cache capacity.

Monitor GC pauses to validate heap sizing

Garbage collection pauses are the canary in the coal mine for heap pressure. When heap is properly sized, GC pauses should be sub-second and happen infrequently—perhaps once every few seconds of sustained load. If you're seeing pauses every few hundred milliseconds or pauses lasting more than two seconds, heap is under pressure and your configuration needs adjustment.

Enable GC logging to track pause duration and frequency:

# In jvm.options
-Xlog:gc*:file=logs/gc.log:time,level,tags:filecount=32,filesize=64m

Then monitor the gc.log output. Look for patterns like full GC collections happening during normal query load—a sign that heap is nearly full and the collector is desperately trying to free space. If full GC pauses coincide with query latency spikes, you have your culprit.

Parse GC logs with tools like gceasy.io or analyze them manually:

# Example GC log line: young generation collection
[2024-01-15T14:22:10.456+0000][0.234s][info ][gc] GC(45) Pause Young 
(Normal) (G1 Evacuation Pause) 256M->128M(512M) 0.145ms

# Example GC log line: full collection (bad sign)
[2024-01-15T14:22:15.789+0000][5.567s][warning][gc] GC(201) Pause Full 
(System.gc()) 480M->456M(512M) 2341ms

A healthy cluster shows young generation collections every few seconds with pause times under 100ms. Full collections should be rare—once a day or less. If you're seeing full GC multiple times per hour, reduce heap or increase node count to redistribute load.

Also monitor the jvm.mem.* metrics from Elasticsearch API:

GET /_nodes/stats/jvm

{
  "nodes": {
    "node-1": {
      "jvm": {
        "mem": {
          "heap_used_in_bytes": 26843545600,
          "heap_max_in_bytes": 34359738368,
          "heap_used_percent": 78,
          "young_gen_used_in_bytes": 536870912,
          "old_gen_used_in_bytes": 26306674688
        }
      }
    }
  }
}

If heap_used_percent regularly climbs above 85%, you're approaching the danger zone. Above 95%, full GC becomes unavoidable and pauses will impact latency. The safe operating range is 50–75% heap utilization under peak load.

Common mistakes

Mistake 1: Setting Xms and Xmx to different values. If you set -Xms20g -Xmx64g, the JVM starts with 20 GB and expands as needed. Each expansion triggers a GC pause. Set both values identically so the heap is pre-allocated and stable from startup.

Mistake 2: Ignoring CompressedOops and pushing heap to 48–62 GB. The 31 GB threshold is hard. Cross it and you lose pointer compression, wasting 30% of your heap on reference overhead. Split into multiple smaller nodes instead.

Mistake 3: Maxing out heap to avoid "out of memory" errors. OOM errors mean you have too much data for your cluster capacity. The fix is more nodes and shards, not bigger heap. Oversized heap just changes the symptom from OOM crashes to GC pauses and query timeouts.

Mistake 4: Not leaving room for off-heap memory. Elasticsearch uses off-heap buffers for networking, Lucene, and the OS itself. Even if heap is half your RAM, the kernel still needs free pages. Aim for 5–10 GB of truly free (unallocated) memory per node for system overhead.

Mistake 5: Fire-and-forget configuration. Set heap once and assume it's right forever. As your data grows and query patterns change, optimal heap size changes. Revisit GC logs and cache behavior quarterly. What worked for 100 GB of data may not work for 1 TB.

Key takeaways

• Allocate heap to at most half of available RAM; the rest fuels the file system cache and improves query performance dramatically.
• Stay under 31 GB heap per node to keep CompressedOops enabled and reduce memory overhead. If you need more capacity, use multiple smaller nodes.
• The file system cache is not overhead—it's the primary performance lever. Without it, every query beyond heap capacity hits disk and latency explodes.
• Monitor GC pause duration and frequency. Young generation pauses under 100ms are normal; full collection pauses longer than 500ms signal heap pressure.
• Keep heap utilization between 50–75% under peak load. Above 85%, full GC becomes frequent and impacts latency. Below 50%, you're over-provisioned and wasting capacity.
• Pre-allocate heap by setting Xms and Xmx identically to avoid runtime resizing and its associated GC pauses.
• Revisit heap sizing quarterly as data volume and query patterns evolve. Configuration that works for 100 GB won't work for 10 TB.