Bulk indexing best practices for Elasticsearch developers

Introduction

Bulk indexing is one of the first challenges every team hits when moving beyond single-document writes in Elasticsearch. Getting it right means the difference between loading a million documents in minutes or watching your cluster grind to a halt under memory pressure. Bulk indexing best practices for Elasticsearch developers aren't optional optimizations—they're the foundation of reliable, fast data ingestion at scale.

In this guide, we'll walk through the concrete decisions and tuning parameters that make bulk operations both fast and safe. Whether you're backfilling historical data, syncing a data warehouse, or pushing real-time events, these techniques apply across the board.

Why bulk indexing matters

Single-document indexing via POST /my-index/_doc has its place, but it's fundamentally inefficient for volume. Each request incurs network overhead, coordination overhead, and forces Elasticsearch to process documents one at a time. Bulk requests allow you to pack multiple operations into a single HTTP call, amortizing that overhead across tens or hundreds of documents.

The tradeoff is that bulk operations require more client-side logic: batching, error handling, retry strategies, and memory management. Done poorly, bulk indexing can overwhelm a cluster faster than single-document writes because you're applying all that pressure at once.

Batches of 5–15 MB: Finding the sweet spot

The most common tuning lever is batch size. Most teams start with a fixed document count—say, 1,000 documents per batch—but that's a trap. A batch of 1,000 tiny 100-byte documents is very different from a batch of 1,000 heavy 50 KB documents. A better approach is to batch by payload size rather than document count.

The 5–15 MB range emerges as a practical default for most workloads. This size balances:

• Memory usage on the client and cluster: small enough that you're not buffering gigabytes of data in your application.
• Network efficiency: large enough that the request/response cycle is amortized effectively.
• Cluster responsiveness: small enough that a single batch failure or slowness doesn't stall your entire pipeline.

Start at 10 MB and measure. If you see the cluster responding quickly and your indexing rate is stable, you might push to 15 MB. If the cluster shows signs of strain—slow refreshes, memory pressure, or timeouts—back off to 7 MB.

Here's a practical approach in pseudocode:

class BulkBatcher {
  buffer = []
  bufferSizeBytes = 0
  maxBatchSizeBytes = 10 * 1024 * 1024  // 10 MB

  addDocument(doc) {
    docJson = JSON.stringify(doc)
    docBytes = Buffer.byteLength(docJson, 'utf8')
    
    if (bufferSizeBytes + docBytes > maxBatchSizeBytes) {
      flush()
    }
    
    buffer.push(doc)
    bufferSizeBytes += docBytes
  }

  flush() {
    if (buffer.empty?) return
    
    bulkPayload = formatBulkRequest(buffer)
    sendToBulk(bulkPayload)
    
    buffer = []
    bufferSizeBytes = 0
  }
}

The key insight: measure your document size distribution in your actual dataset, then set batch limits based on bytes, not count.

Parallel bulk requests: Saturate the cluster safely

Once you've nailed batch size, the next lever is concurrency. Sending bulk requests one at a time leaves CPU and I/O on the table. Most clusters can handle 4–8 concurrent bulk requests comfortably; beyond that, you risk queue saturation and memory pressure.

The pattern is simple: maintain a "flight" of in-flight requests. As one completes, start the next.

class ParallelBulkIndexer {
  maxConcurrent = 6
  inFlightCount = 0
  pendingBatches = queue()

  indexBatch(batch) {
    pendingBatches.push(batch)
    processNext()
  }

  processNext() {
    while (inFlightCount < maxConcurrent && !pendingBatches.empty?) {
      batch = pendingBatches.pop()
      inFlightCount++
      
      client.bulk(batch)
        .then(response => {
          handleResponse(response)
          inFlightCount--
          processNext()
        })
        .catch(error => {
          handleError(batch, error)
          inFlightCount--
          processNext()
        })
    }
  }
}

Start with maxConcurrent = 4. Monitor cluster metrics (CPU, JVM heap, thread pool queue depth). If CPU is under 70% and heap under 80%, try 6 or 8. If you see queues backing up or timeouts, dial back to 2 or 3.

In production systems handling millions of documents per day, we typically run 4–6 concurrent streams. Any higher and you're fighting Elasticsearch's own resource limits rather than using them efficiently.

Increase refresh_interval during big loads

By default, Elasticsearch refreshes its in-memory buffers every 1 second, making newly indexed documents searchable. That's great for real-time use cases, but during bulk loading it's wasteful. Refreshes cost CPU and I/O; if you're indexing millions of documents in an hour, you're refreshing thousands of times unnecessarily.

Before bulk loading, disable or dramatically increase the refresh interval:

// Before bulk indexing
PUT /my-index/_settings
{
  "index.refresh_interval": "-1"
}

// ... perform all bulk indexing ...

// After bulk indexing completes
PUT /my-index/_settings
{
  "index.refresh_interval": "1s"
}

// Explicitly refresh once
POST /my-index/_refresh

Setting refresh_interval to -1 disables automatic refresh entirely. This can cut bulk load time by 30–50% for large datasets because you're not paying the refresh cost on every flush. Just be aware: documents won't be searchable until you manually refresh at the end.

An alternative, if you need documents to be searchable sooner, is to increase the interval to something like 30s or 60s:

PUT /my-index/_settings
{
  "index.refresh_interval": "30s"
}

This is a good middle ground for backfills that take several hours: you get substantial performance gains without making documents completely invisible during the load.

One more consideration: if you have replicas, disabling refresh affects the replica shards too. For critical indices, you might consider temporarily reducing the replica count to 0 during bulk loads, then rebuilding replicas afterward. Just be aware of the tradeoffs in availability.

Check per-item errors in responses

Here's where many teams drop the ball. Bulk requests return a 200 OK even if half the documents failed to index. The HTTP status code tells you nothing about individual operation success.

Every bulk response includes an items array with an entry for each operation. You must inspect it:

{
  "took": 145,
  "errors": true,
  "items": [
    {
      "index": {
        "_index": "my-index",
        "_id": "1",
        "_version": 1,
        "result": "created",
        "_shards": {
          "total": 2,
          "successful": 2,
          "failed": 0
        },
        "_seq_no": 0,
        "_primary_term": 1,
        "status": 201
      }
    },
    {
      "index": {
        "_index": "my-index",
        "_id": "2",
        "error": {
          "type": "mapper_parsing_exception",
          "reason": "failed to parse field [timestamp] of type [date]...",
          "caused_by": {
            "type": "illegal_argument_exception",
            "reason": "Invalid format: \"not-a-date\"..."
          }
        },
        "status": 400
      }
    }
  ]
}

In this response, document 1 succeeded (status 201), but document 2 failed with a parsing error. If you ignore the items array, you've silently lost data.

Process every response like this:

function processBulkResponse(response, originalBatch) {
  failedItems = []
  successCount = 0
  
  response.items.forEach((item, index) => {
    operation = Object.keys(item)[0]  // "index", "create", "update", "delete"
    opResult = item[operation]
    
    if (opResult.error) {
      failedItems.push({
        document: originalBatch[index],
        error: opResult.error,
        status: opResult.status
      })
    } else {
      successCount++
    }
  })
  
  if (failedItems.length > 0) {
    logger.warn(`Bulk request: ${successCount} succeeded, ${failedItems.length} failed`)
    handleFailedItems(failedItems)  // retry, log, alert, etc.
  }
  
  return {
    successCount: successCount,
    failedCount: failedItems.length,
    failures: failedItems
  }
}

Common error patterns you'll see:

• mapper_parsing_exception: Document field doesn't match the mapping type. Validate data shape before indexing.
• version_conflict_engine_exception: You tried to update a document with an outdated version. Handle idempotency carefully.
• illegal_argument_exception: Malformed request body or invalid field value. Check your bulk request formatting.
• Circuit breaker errors: Cluster is under memory pressure. Back off concurrency and batch size.

For transient errors (circuit breaker, timeouts), retry the failed batch. For permanent errors (parsing, schema mismatch), log and skip, then investigate the root cause in your data pipeline.

Common mistakes

Ignoring the errors flag and per-item status codes: This is the biggest one. You think you've indexed 1 million documents, but 50,000 silently failed. Always inspect the response.

Batching purely by document count: A batch of 10,000 tiny documents and a batch of 10,000 heavy documents will have wildly different performance profiles. Batch by payload size instead.

Setting refresh_interval to -1 and forgetting to re-enable it: Your index will be unsearchable until you manually refresh. Document this step in your runbook and automate it if possible.

Too much concurrency without monitoring: Throwing 20 concurrent requests at a modest cluster will cause memory exhaustion and cascading failures. Start conservative and measure.

Not handling backpressure: If your source system is producing documents faster than Elasticsearch can ingest them, your application will consume unbounded memory. Use queues with fixed capacity and apply backpressure to the source.

Retrying failed batches indefinitely: Some documents will never succeed (invalid schema, corrupt data). After a reasonable retry count, move them to a dead-letter queue and investigate.

Monitoring and tuning in production

During bulk loads, watch these metrics on your Elasticsearch cluster:

• Indexing rate (docs/sec): How many documents per second are you actually indexing? This is your throughput, the number you care about most.
• Bulk queue depth: Check GET _nodes/stats/thread_pool/bulk. If the queue is consistently full, reduce concurrency or batch size.
• JVM heap usage: If it's creeping toward the limit, reduce batch size and concurrency.
• CPU usage: If you're under 70%, you have headroom to increase concurrency or batch size.
• Segment merge activity: Monitor GET _stats and watch for large merges that might cause latency spikes.

A healthy bulk load on a 3-node cluster with 16 GB heap per node typically achieves 10,000–50,000 docs/sec, depending on document size and your hardware. If you're consistently lower, profiling your bottleneck (network, CPU, I/O, or Elasticsearch itself) is your next step.

Key takeaways

• Batch by payload size (5–15 MB), not document count. Measure your actual document sizes and adjust.
• Use 4–6 concurrent bulk requests to saturate the cluster without overwhelming it. Monitor queue depth and adjust based on metrics.
• Disable or increase refresh_interval during bulk loads to avoid wasting CPU on unnecessary refreshes. Remember to re-enable and manually refresh when done.
• Always inspect the items array in bulk responses. HTTP 200 doesn't mean all documents succeeded.
• Implement a retry strategy for transient failures (timeouts, circuit breakers) and a dead-letter queue for permanent failures (schema mismatches).
• Monitor indexing rate, bulk queue depth, heap usage, and CPU during production loads. Use these signals to tune batch size and concurrency.