Handling deletes in Elasticsearch: a step-by-step guide

Understanding how Elasticsearch handles deletion

Deletes in Elasticsearch are not the instant cleanup operations you might expect from a relational database. When you delete a document, Elasticsearch doesn't immediately reclaim storage or remove it from disk. Instead, documents are marked for deletion and physically removed only during segment merges. Understanding this behavior is essential when designing systems that rely on frequent deletions, compliance workflows, or large-scale data removal. This guide walks you through handling deletes in Elasticsearch: a step-by-step guide, covering the mechanisms, trade-offs, and best practices we've learned from production experience.

How deleted documents are actually stored and cleaned up

When you send a delete request to Elasticsearch, the document isn't erased from the underlying Lucene segments immediately. Instead, Elasticsearch writes a deletion marker—a tombstone—into the index. During search queries, Elasticsearch filters out documents marked as deleted so they don't appear in results. The physical removal happens only when Lucene merges segments together.

This lazy-deletion model exists for performance reasons. Rewriting segments on every delete would be prohibitively expensive. Instead, merges happen periodically (controlled by merge policies), and deleted documents are truly purged at that time. Until a merge occurs, deleted documents still consume disk space.

You can inspect this behavior by checking segment information:

GET my-index/_segments

The response includes num_search_segments and num_deleted_docs. If you see a high count of deleted docs and many small segments, your index hasn't merged recently, meaning deletions haven't reclaimed space yet.

For a single-document delete, this is rarely a concern. Problems emerge when you delete thousands or millions of documents regularly. The accumulation of deletion markers can bloat your index, increase memory usage, and slow down queries as Elasticsearch filters more tombstones.

Using delete by query for bulk deletes

When you need to remove many documents at once—say, purging old logs or removing user data after account closure—the Delete by Query API is more efficient than deleting documents individually.

Here's a practical example: imagine you're running an analytics index and want to remove all events older than 90 days.

POST my-events-index/_delete_by_query
{
  "query": {
    "range": {
      "timestamp": {
        "lt": "now-90d"
      }
    }
  }
}

This single request deletes all matching documents in a single pass. Elasticsearch batches the work internally, executing the query to find candidates, then marking them for deletion.

By default, Delete by Query processes 1000 documents per batch. You can tune this with the scroll_size parameter:

POST my-events-index/_delete_by_query?scroll_size=5000
{
  "query": {
    "range": {
      "timestamp": {
        "lt": "now-90d"
      }
    }
  }
}

A larger scroll size can be faster for very large deletes, but it consumes more memory. Start with 5000 and adjust based on your cluster's resources.

One critical consideration: Delete by Query holds read locks on the index during execution. If your application is heavily writing to the same index, performance will suffer. We recommend running bulk deletes during off-peak hours or on a dedicated index that you're prepared to pause writes on temporarily.

You can also use a query filter to be precise about what you're deleting:

POST my-events-index/_delete_by_query
{
  "query": {
    "bool": {
      "must": [
        { "match": { "status": "spam" } },
        { "range": { "created_at": { "lt": "2024-01-01" } } }
      ]
    }
  }
}

Always test your query first using the Search API to see exactly which documents will be deleted.

Using Index Lifecycle Management (ILM) to delete entire indexes

The most efficient way to delete data at scale is not to delete individual documents at all—it's to delete entire indexes. If your data naturally partitions by time (logs, events, metrics), Index Lifecycle Management lets you automate index deletion based on age.

A typical pattern is to create daily or weekly indexes. For example, logs might live in indexes named logs-2024-01-15, logs-2024-01-16, and so on. With ILM, you define a policy that deletes indexes older than a retention period.

Here's an example policy that keeps indexes for 30 days:

PUT _ilm/policy/logs-policy
{
  "policy": {
    "phases": {
      "hot": {
        "min_age": "0d",
        "actions": {
          "rollover": {
            "max_primary_shard_size": "50GB",
            "max_age": "1d"
          }
        }
      },
      "warm": {
        "min_age": "1d",
        "actions": {
          "set_priority": {
            "priority": 50
          }
        }
      },
      "delete": {
        "min_age": "30d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Once this policy is attached to an index template, Elasticsearch automatically rolls over old indexes to a "warm" phase and deletes them after 30 days. No manual intervention required.

This approach avoids the problems of accumulating deletion markers because entire segments—and the index itself—are removed. It's far faster and more reliable than trying to delete millions of individual documents.

In production, we've seen ILM reduce storage costs by 60–70% compared to systems that attempted large bulk deletes on monolithic indexes. Time-series data is ideal for this pattern.

Why frequent large deletes hurt performance

Running large delete operations repeatedly on the same index creates several problems:

• Deletion marker bloat: Each deleted document adds a tombstone that must be checked during every query, slowing down search performance even after the physical delete.
• Segment fragmentation: Frequent deletes fragment your index into many small segments, increasing memory overhead and slowing merges.
• Disk I/O spikes: Merging segments to reclaim space is I/O-intensive. Scheduling large merges during traffic peaks can starve other queries of resources.
• Lock contention: Delete by Query holds read locks. Concurrent writes block, causing latency spikes for your application.

Instead of deleting frequently, consider these alternatives:

• Use time-based indexes and ILM to delete by whole index.
• Keep deletion periods infrequent and off-peak (nightly or weekly batches).
• For data that changes state rather than gets deleted (e.g., marking as "archived" rather than removing), use updates instead of deletes.
• For large-scale purges (GDPR right-to-be-forgotten), consider a dedicated purge index or reindex-and-filter approach instead of Delete by Query.

Common mistakes

Mistake 1: Assuming deletes free space immediately. They don't. Disk space is reclaimed only during segment merges, which can happen hours or days later. If you're under storage pressure, forcing a merge won't help if you've only deleted documents recently.

Mistake 2: Running Delete by Query on production indexes during peak traffic. The read locks will block writes and cause latency spikes. Always test on a staging index first and schedule bulk deletes during maintenance windows.

Mistake 3: Deleting without a well-tested query. One typo in your delete query can wipe out data you meant to keep. Always run your query through the Search API first, inspect the results, and verify the count matches your expectations.

Mistake 4: Not cleaning up after bulk deletes. After a large delete operation, force a merge to reclaim space and improve query performance:

POST my-index/_forcemerge?max_num_segments=1

This is expensive and should be done off-peak, but it's the only way to truly reclaim disk space after bulk deletes.

Mistake 5: Using Delete by Query for routine data cleanup instead of ILM. If your data is time-partitioned, ILM is vastly more efficient. Reserve Delete by Query for ad-hoc purges and compliance requests.

Key takeaways

• Deleted documents are marked with tombstones and removed only during segment merges; they consume disk space until merges occur.
• Delete by Query is efficient for bulk deletes but holds read locks; schedule it during off-peak hours and test queries first.
• Index Lifecycle Management is the best approach for time-series data; delete entire indexes instead of individual documents.
• Frequent large deletes fragment indexes and create query performance problems; use time-based indexes or state flags instead where possible.
• After bulk deletes, force a merge to reclaim disk space and improve performance, but only during maintenance windows.
• Always verify your delete query with a search first; deleting is permanent.