The challenge of changing Elasticsearch mappings
Once you've indexed millions of documents in Elasticsearch, you'll discover that mappings aren't as flexible as you'd hoped. A field type needs to change, a new analyzer is required, or a field must be renamed—and reindexing the entire dataset during normal business hours isn't an option. This guide walks you through zero-downtime reindexing: a hands-on guide for engineers who need to modify mappings safely without dropping queries or accepting downtime.
The core challenge is that Elasticsearch doesn't allow you to modify field mappings in place. The solution is a four-step process: create a new index with the updated mapping, copy data from the old index to the new one, catch up on writes that arrived during the copy, and atomically switch traffic over using aliases. Let's build this step by step.
Step 1: Create a new index with the new mapping
Before you touch any production data, create a second index with your desired mapping changes. The key is to use an alias so your application continues writing to a logical name rather than a concrete index name.
Assume your current setup has an index called products and an alias products-live that points to it. You want to add a new field type and change an analyzer. First, create the new index:
PUT /products-v2
{
"settings": {
"number_of_shards": 3,
"number_of_replicas": 1,
"analysis": {
"analyzer": {
"custom_text": {
"type": "standard",
"stopwords": "_english_"
}
}
}
},
"mappings": {
"properties": {
"id": { "type": "keyword" },
"name": {
"type": "text",
"analyzer": "custom_text",
"fields": {
"keyword": { "type": "keyword" }
}
},
"description": { "type": "text" },
"category": { "type": "keyword" },
"created_at": { "type": "date" },
"stock_level": { "type": "integer" },
"last_updated": { "type": "date" }
}
}
}
Notice we've named it products-v2. This naming convention makes it easy to track versions and clean up old indices later. The settings and mappings should mirror your current index, with only the specific changes you need applied. If you're unsure about shard count, match your current index or consult your cluster's capacity planning.
Step 2: Reindex the data
With both indices in place, copy all existing documents from the old index to the new one. Elasticsearch's Reindex API handles this efficiently, and you can control the batch size and throttling to avoid overwhelming your cluster.
POST /_reindex
{
"source": {
"index": "products"
},
"dest": {
"index": "products-v2"
},
"slices": 5
}
The slices parameter parallelizes the reindex operation across five slices, which speeds up the copy significantly on multi-node clusters. For very large indices (hundreds of millions of documents), monitor cluster health and consider reducing slices if you see elevated CPU or memory usage.
The Reindex API is synchronous by default, so if your dataset is enormous, you can run it asynchronously and check status:
POST /_reindex?wait_for_completion=false
{
"source": {
"index": "products"
},
"dest": {
"index": "products-v2"
},
"slices": 5
}
This returns a task ID. Check progress with GET /_tasks?detailed=true&actions=*reindex. In production, we typically let reindex run to completion on a less-busy time window (late evening or early morning) to minimize impact on read latency.
Step 3: Catch up on writes made during the copy
Here's the critical insight: while reindex ran, your application kept writing to The challenge is determining what changed. If your documents have a Replace the timestamp with the exact moment the first reindex began. This approach works well if all your documents are timestamped. For systems without reliable timestamps, consider using change data capture (CDC) or maintaining a changelog table to track modifications. A more robust pattern in high-throughput systems is to dual-write during the catch-up phase: your application writes to both the old index and the new index simultaneously for a short window. This guarantees no writes are missed. Then, once you're confident the new index has all data, you switch traffic over. Now both indices contain the same data. The final step is to point the alias from the old index to the new one. This is atomic at the Elasticsearch level, so no queries are dropped: All subsequent reads and writes to Once you're satisfied, delete the old index: This frees up disk space immediately. If you want to keep a safety window in case you need to rollback, keep the old index around for a day or two before deleting.products-live. New documents or updates arrived that aren't yet in products-v2
last_updated timestamp, you can use a range query:POST /_reindex
{
"source": {
"index": "products",
"query": {
"range": {
"last_updated": {
"gte": "2024-01-15T14:32:00Z"
}
}
}
},
"dest": {
"index": "products-v2"
}
}
Step 4: Swap the alias atomically
POST /_aliases
{
"actions": [
{ "remove": { "index": "products", "alias": "products-live" } },
{ "add": { "index": "products-v2", "alias": "products-live" } }
]
}
products-live go to products-v2. Your application code doesn't change; it continues querying and indexing against the alias. After the switch, monitor error rates and latency for a few minutes to confirm everything is working.DELETE /products
Common mistakes
Key takeaways
_aliases API, then delete the old index once you're confident in the new setup.
Comments
Loading comments…