Indexing documents in Elasticsearch & Search Engineering

Indexing documents in Elasticsearch & Search Engineering

By day ten of building a search system, you've got the cluster running and mappings defined. Now comes the work that makes it all real: putting your data into Elasticsearch. Indexing documents is where theory meets practice, and how you do it shapes everything from query latency to operational burden. Let's walk through the practical patterns we've learned across production systems—from single-document operations to bulk ingestion strategies that handle millions of records without melting your cluster.

Index one document — PUT or POST to the index

The simplest path to getting data into Elasticsearch is a single document request. You hit the index endpoint with either PUT or POST, send your JSON payload, and Elasticsearch stores it. The difference is subtle but matters: PUT requires an explicit ID in the URL (you control the document ID), while POST lets Elasticsearch generate one for you. In production, we usually reach for PUT so we can tie documents to stable identifiers in our source system.

Here's a basic example against a products index:

PUT /products/_doc/1
{
  "name": "Ceramic mug",
  "price": 12.5,
  "tags": ["kitchen"],
  "sku": "MUG-001",
  "stock": 150
}

Elasticsearch responds with metadata about the operation:

{
  "_index": "products",
  "_type": "_doc",
  "_id": "1",
  "_version": 1,
  "result": "created",
  "_shards": {
    "total": 2,
    "successful": 2,
    "failed": 0
  },
  "_seq_no": 0,
  "_primary_term": 1
}

That response tells you the document landed on both the primary shard and its replica. The version number (1) tracks how many times this document has been modified. The seq_no and primary_term are internal bookkeeping for Elasticsearch's consensus model—useful for optimistic locking when you need it.

For indexing a single product update from your backend when stock changes, PUT works fine:

PUT /products/_doc/1
{
  "name": "Ceramic mug",
  "price": 12.5,
  "tags": ["kitchen"],
  "sku": "MUG-001",
  "stock": 145
}

Notice we sent the entire document again. That's important: a PUT replaces the whole thing. If you only wanted to change stock and left it out, Elasticsearch would delete that field. For partial updates, we'll cover that in a moment—that's where _update comes in.

Bulk API — Index thousands of documents efficiently

Single requests are fine for one product. But when you're syncing ten thousand SKUs from your catalog or replaying a data migration, firing individual requests is slow and wasteful. That's where the Bulk API saves us. Instead of one document per request, you pack hundreds or thousands into a single call, drastically reducing network overhead and request parsing.

The Bulk API uses newline-delimited JSON (NDJSON). Each line is either an action (index, create, update, delete) followed by a newline, then the document payload on the next line. No pretty-printing, no trailing commas—just compact JSON on each line.

Here's a realistic bulk request indexing multiple products:

POST _bulk
{ "index": { "_index": "products", "_id": "1" } }
{ "name": "Ceramic mug", "price": 12.5, "tags": ["kitchen"], "sku": "MUG-001", "stock": 150 }
{ "index": { "_index": "products", "_id": "2" } }
{ "name": "Stainless steel water bottle", "price": 24.99, "tags": ["outdoor", "sports"], "sku": "BOTTLE-002", "stock": 340 }
{ "index": { "_index": "products", "_id": "3" } }
{ "name": "Bamboo cutting board", "price": 18.0, "tags": ["kitchen"], "sku": "BOARD-003", "stock": 87 }
{ "create": { "_index": "products", "_id": "4" } }
{ "name": "Cast iron skillet", "price": 45.5, "tags": ["kitchen", "cooking"], "sku": "SKILLET-004", "stock": 42 }

Notice we used both index and create actions. Index means "insert or replace," while create means "only insert if it doesn't exist"—useful when you want to avoid overwriting existing documents. If you're doing a full catalog sync, index is the right choice. If you're only adding new products, create prevents accidentally wiping out enriched data you added manually.

The response from a bulk request is a single JSON object containing an array of results, one per operation:

{
  "took": 45,
  "errors": false,
  "items": [
    {
      "index": {
        "_index": "products",
        "_id": "1",
        "_version": 1,
        "result": "created",
        "_shards": { "total": 2, "successful": 2, "failed": 0 }
      }
    },
    {
      "index": {
        "_index": "products",
        "_id": "2",
        "_version": 1,
        "result": "created",
        "_shards": { "total": 2, "successful": 2, "failed": 0 }
      }
    }
  ]
}

Check that errors field. If it's true, dig through the items array to find which operations failed. One document failing doesn't stop the rest—the API continues and tells you what broke. In a real sync pipeline, we log failures separately and retry them later.

A production tip: the Bulk API respects your cluster's resources. If you send five million documents at once, you'll likely hit memory limits or cause GC pauses. Instead, batch them into chunks of 1,000 to 10,000 documents per bulk request, depending on document size and cluster capacity. A 5 MB document payload per request is a reasonable ballpark. We've had good results sending bulk requests in parallel—say, five concurrent bulk operations—so long as you monitor cluster health and back off if indexing latency climbs.

Refresh interval — Documents become searchable after a refresh

Here's a gotcha we've all hit: you index a document, immediately run a search, and it doesn't show up. That's because Elasticsearch doesn't make new documents searchable instantly. Internally, Elasticsearch uses an inverted index structure called a segment. Newly indexed documents sit in a buffer called the refresh buffer. To make them searchable, Elasticsearch must flush that buffer and build a new segment—an operation called a refresh.

By default, Elasticsearch refreshes every one second. That's why, if you wait a moment and search again, your new document appears. For most use cases, one-second latency is acceptable and healthy. It batches writes together, reducing segment creation overhead and keeping search performance stable.

You can control the refresh interval per index. Here's setting it to five seconds when creating an index:

PUT /products
{
  "settings": {
    "index.refresh_interval": "5s"
  },
  "mappings": {
    "properties": {
      "name": { "type": "text" },
      "price": { "type": "float" },
      "tags": { "type": "keyword" }
    }
  }
}

Or update it on an existing index:

PUT /products/_settings
{
  "index.refresh_interval": "5s"
}

Why would you change it? During bulk indexing, longer refresh intervals reduce overhead. If you're ingesting millions of documents, setting refresh to 30 seconds or even -1 (disable automatic refresh) during the bulk load, then resetting it to 1s afterward, cuts indexing time significantly. The tradeoff is that newly indexed data won't appear in searches until you manually refresh or the interval elapses.

For real-time search requirements—think order status or chat messages—you might set refresh to 100ms or even force an immediate refresh after critical writes:

POST /orders/_refresh

That command refreshes the orders index immediately, making all pending documents searchable. Use it sparingly. Refreshing is I/O intensive, and doing it after every single write defeats the purpose of batching.

Updates — Partial updates rewrite the document internally

As your product catalog evolves, you rarely want to replace entire documents. Stock changes, prices drop, descriptions get refined. Elasticsearch offers the _update API for partial updates. Instead of sending the full document, you send only the fields that changed:

POST /products/_update/1
{
  "doc": {
    "stock": 145,
    "price": 11.99
  }
}

Elasticsearch retrieves the existing document, merges in your changes, and stores the updated version. The response includes the new version number:

{
  "_index": "products",
  "_id": "1",
  "_version": 2,
  "result": "updated"
}

Internally, _update still rewrites the entire document on disk—Elasticsearch doesn't support true in-place updates at the storage level. But from your perspective, you're only sending the delta, which saves bandwidth and makes your code cleaner.

If you want to conditionally update only if the document hasn't changed since you last saw it, use the optimistic locking fields:

POST /products/_update/1?if_seq_no=0&if_primary_term=1
{
  "doc": {
    "stock": 145
  }
}

If someone else updated that document and seq_no changed, your request fails with a 409 conflict. This pattern prevents lost updates in concurrent scenarios.

You can also run a script during an update. If your business logic needs to decrement stock atomically:

POST /products/_update/1
{
  "script": {
    "source": "ctx._source.stock -= params.amount",
    "params": {
      "amount": 5
    }
  }
}

Scripts are powerful but slower than bulk updates—reserve them for cases where the logic is genuinely complex or must be atomic. For high-throughput scenarios, fetch the document in your application, compute the new value, and send a bulk update instead.

Common mistakes

• Forgetting the action line in bulk requests: Every document in a bulk call needs an action (index, create, update, delete) on its own line. A missing action line breaks parsing and silently discards documents. Always validate your NDJSON format before sending.
• Setting refresh interval too aggressively: Refreshing every 100ms during high-volume indexing causes excessive segment creation, GC pressure, and actually slows down your cluster. Leave the default or extend it during bulk operations.
• Sending complete documents with _update: Using _update with the full document works, but it's inefficient. Use _update for partial changes; use PUT when you need to replace everything.
• Ignoring bulk request errors: If a bulk response has "errors": true, some operations failed. Skipping the error check means data inconsistency. Always log and retry failures.
• Indexing without a stable ID: Letting Elasticsearch generate IDs via POST makes deduplication hard. Use PUT with an ID from your source system (product SKU, order ID, user ID) so re-indexing the same data doesn't create duplicates.

Key takeaways

• Use PUT with an explicit document ID to tie Elasticsearch documents to your source system and avoid duplicates.
• The Bulk API is your friend for ingesting thousands of documents—batch them into 1,000-10,000 document chunks and consider parallel requests.
• Documents become searchable after a refresh; the default one-second interval is usually right, but extend it during bulk loads and disable it afterward for speed.
• _update sends only changed fields, but Elasticsearch still rewrites the full document internally—use it for clarity, not for performance optimization.
• Always validate bulk responses for errors, log failures, and implement retry logic; one failed operation doesn't stop the rest.
• For concurrent safety, use optimistic locking with if_seq_no and if_primary_term to prevent lost updates.