Versioning and concurrency: patterns, pitfalls and fixes

Opening: The race condition trap

Concurrent writes in distributed search systems are deceptively dangerous. Two requests arrive within milliseconds of each other, both read the same document, both make changes, and one silently overwrites the other. This is the versioning and concurrency problem: patterns, pitfalls and fixes that separate production systems from chaos.

In Elasticsearch, you can lose data without a single error message. Your application thinks both updates succeeded. Your users see stale information. The search index diverges from your source of truth. Let's walk through the patterns that prevent this, the mistakes we've made in production, and how to fix them.

Understanding if_seq_no and if_primary_term

Elasticsearch ships with built-in optimistic locking via sequence numbers and primary terms. Every document has two metadata fields that change when the document is modified:

• _seq_no: an incrementing counter that increases with every write to that document
• _primary_term: changes when the primary shard moves to a different node (e.g., after a failure)

These form an immutable version tuple. When you read a document, you get its current _seq_no and _primary_term. If you want to write it back, you can tell Elasticsearch: "only write this if the current _seq_no and _primary_term still match what I saw." If they don't match, someone else changed the document since you read it.

Here's a typical flow:

1. Read a product document to get its price and stock
2. Elasticsearch returns _seq_no: 15, _primary_term: 1
3. Your code calculates new stock after a purchase
4. You update with if_seq_no=15&if_primary_term=1
5. If concurrent write happened: Elasticsearch rejects your update with 409 Conflict

The beauty is that you choose when to apply this. It's not automatic—it's opt-in protection.

PUT /products/_doc/SKU-001?if_seq_no=15&if_primary_term=1
{
  "name": "Widget Pro",
  "price": 29.99,
  "stock": 47
}

If another client updated the same document between your read and write, Elasticsearch returns 409 Conflict and the update fails. Your application must then retry: read the latest version, recalculate, and try again.

External versioning from your database

Sometimes the authoritative version number doesn't live in Elasticsearch. It lives in your PostgreSQL, MongoDB, or DynamoDB. Your Elasticsearch index is a replica, denormalized for search speed. In that case, you should sync the version from your source database into Elasticsearch.

A common pattern: store a db_version field in every document that tracks the version from your source system.

PUT /orders/_doc/ORDER-987
{
  "customer_id": "CUST-123",
  "total": 159.99,
  "status": "shipped",
  "db_version": 3,
  "updated_at": "2024-01-15T10:22:30Z"
}

Your application flow then becomes:

1. Read order from PostgreSQL (includes its version, say 3)
2. Modify it locally
3. Update PostgreSQL with WHERE version = 3 (your database's optimistic lock)
4. On success, update Elasticsearch with the new db_version (now 4)
5. If PostgreSQL rejects due to version mismatch, retry from step 1; don't touch Elasticsearch

This approach keeps your Elasticsearch version in sync with the source of truth. It also means your Elasticsearch writes are idempotent: if the same version update is retried, Elasticsearch just overwrites with the same data. No harm done.

The downside: you need a separate version field to track. The upside: your Elasticsearch layer stays consistent with reality, even if a write fails partway through or is retried.

Conflicts return 409: recognizing and handling them

When you use if_seq_no and if_primary_term, a concurrent write triggers a 409 Conflict response. Your HTTP client receives it immediately. You must handle it gracefully.

A naive approach: retry instantly. Wrong. If multiple clients are contending, you'll hammer the cluster and likely collide again immediately. Better approach: exponential backoff with jitter.

{
  "error": {
    "type": "version_conflict_engine_exception",
    "reason": "[1]: version conflict, current version [16] is different than the one provided [15]",
    "index": "products",
    "cause": {
      "type": "version_conflict_engine_exception",
      "reason": "[1]: version conflict, current version [16] is different than the one provided [15]"
    }
  },
  "status": 409
}

Recognize this response by its status code (409) and type (version_conflict_engine_exception). In your client code:

// Pseudocode: retry logic for 409 conflicts
function updateWithRetry(docId, updates, maxRetries = 5) {
  for (let attempt = 0; attempt < maxRetries; attempt++) {
    try {
      const doc = readDocument(docId);
      const updated = applyUpdates(doc, updates);
      const result = updateDocument(docId, updated, {
        if_seq_no: doc._seq_no,
        if_primary_term: doc._primary_term
      });
      return result;
    } catch (error) {
      if (error.status === 409 && attempt < maxRetries - 1) {
        const backoffMs = Math.pow(2, attempt) * 100 + Math.random() * 100;
        await sleep(backoffMs);
        continue;
      }
      throw error;
    }
  }
}

The key insight: when you get 409, you must re-read the document. The _seq_no has changed. Your old version tuple is now stale. Retrying with the same tuple will fail again.

Retry with fresh data: the complete cycle

A production-ready retry cycle for high-concurrency scenarios looks like this:

1. Read: fetch the document and capture _seq_no, _primary_term, and current data
2. Calculate: apply your business logic to compute the new state
3. Write with version check: attempt update with if_seq_no and if_primary_term
4. Handle 409: if conflict, wait with backoff, go to step 1
5. Log and alert: if max retries exceeded, log the failure and alert on-call

The retry loop should be bounded. In our experience, 3–5 retries with exponential backoff (200ms, 400ms, 800ms, 1.6s, 3.2s) handles the vast majority of contention in production. Beyond that, you likely have a bigger problem: thundering herd, too much write pressure, or a hot document getting hammered by many clients.

One more pattern: if you're updating multiple documents as part of a transaction, use bulk operations and version checks on each doc. Elasticsearch transactions don't span multiple documents, so if one bulk update fails with 409, you need to decide: retry the entire bulk, or skip that item and continue. Document your choice clearly in comments and monitoring.

Common mistakes

Mistake 1: Ignoring the 409 response. Some teams see a 409 and treat it as a permanent error, immediately returning 500 to the user. In reality, 409 is transient contention. Retry it. Your user's request will likely succeed on the second attempt.

Mistake 2: Retrying without reading fresh data. After a 409, you must re-read the document before retrying the update. Reusing the old _seq_no and _primary_term will fail again instantly. This wastes latency and increases failure rates.

Mistake 3: Mixing if_seq_no and external versioning carelessly. If you use both, make sure they're in sync. For example, don't update db_version in your source system but forget to update it in Elasticsearch. You'll lose the version check's protection.

Mistake 4: Unlimited retry loops. A buggy retry function can loop forever if the conflict never resolves (e.g., another service is also writing to the same doc constantly). Always set a max retry count and a timeout. Log and alert when you hit it.

Mistake 5: Not testing concurrent scenarios. Versioning bugs are hard to spot in staging with one user. Load-test with 50+ concurrent writers to the same document. Use tools like Apache JMeter or Gatling to simulate real contention.

Key takeaways

• Elasticsearch's _seq_no and _primary_term provide built-in optimistic locking. Use if_seq_no and if_primary_term query parameters to protect writes from concurrent updates.
• A 409 Conflict response means another write succeeded before yours. It's not an error; it's a signal to retry with fresh data.
• Always re-read the document after a 409 before attempting the write again. Never retry with stale version numbers.
• Combine if_seq_no/if_primary_term with exponential backoff and jitter for reliable, bounded retry logic.
• If your source of truth is external (PostgreSQL, etc.), sync the version number into Elasticsearch as a field so both systems stay consistent.
• Set a max retry limit and max timeout. Log and alert when you exceed them; it indicates a systemic issue, not a transient conflict.
• Test concurrency patterns in staging and production-like environments before deploying. Hot-document scenarios expose versioning bugs quickly.