Runtime fields: Elasticsearch best practices explained

Introduction

Runtime fields in Elasticsearch let you compute field values at query time instead of indexing them upfront—a powerful way to explore data without the cost of reindexing. We'll cover how to add fields without reindexing, when runtime fields make sense for exploration, why they trade speed for flexibility, and when you should graduate them to indexed fields for production workloads.

Add fields without reindexing

One of the biggest operational hurdles in Elasticsearch is reindexing. When your mapping changes or you need to add new computed fields, a full reindex can lock resources, delay deployments, and disrupt query performance on large datasets. Runtime fields eliminate this friction entirely.

A runtime field is defined in your mapping but not stored on disk. Instead, the compute logic runs during query execution. This means you can add new derived fields to your mapping, deploy the change instantly, and start querying without touching a single document.

Consider a real-world scenario: you have an e-commerce index with order data including unit_price and quantity. A product team wants to analyze line_total (unit_price × quantity) but you can't reindex millions of orders. With a runtime field, you define the calculation in the mapping and it's immediately available:

{
  "mappings": {
    "runtime": {
      "line_total": {
        "type": "double",
        "script": {
          "source": "emit(doc['unit_price'].value * doc['quantity'].value)"
        }
      }
    }
  }
}

Now any query can reference line_total as if it were a regular field. The mapping change is instant—no reindex, no downtime, no waiting.

This is especially valuable during exploratory phases. When you're not sure if a derived metric will be useful long-term, runtime fields let you validate the idea fast. If the team decides they don't need it, you simply remove it from the mapping. If they love it, you can index it permanently later.

Useful for exploration

Runtime fields shine when you're prototyping, building reports, or testing hypotheses about your data. They're a sandbox for new logic without committing resources to permanent indexed storage.

Imagine a SaaS analytics platform where customers want to segment users by a "lifetime_value" bucket. You might want to test different thresholds—gold tier for >$10k, silver for $1k–$10k, bronze for <$1k. Rather than create three separate indexed fields, you can experiment with a single runtime field:

{
  "mappings": {
    "runtime": {
      "user_tier": {
        "type": "keyword",
        "script": {
          "source": """
            double ltv = doc['lifetime_value'].value;
            if (ltv > 10000) {
              emit('gold');
            } else if (ltv > 1000) {
              emit('silver');
            } else {
              emit('bronze');
            }
          """
        }
      }
    }
  }
}

Your team can run aggregations, filters, and dashboards on user_tier immediately. When the thresholds need tweaking, you update the script and redeploy—no indexing wait.

Runtime fields are also invaluable for multi-source data. If you're combining fields from nested documents or arrays, or transforming string data into structured values, runtime computation lets you defer that work until query time and experiment freely.

They're equally useful for A/B testing. You can define a runtime field that hashes a user ID and assigns them to a cohort, then filter queries by that cohort—all without reindexing or maintaining separate datasets.

Slower than indexed fields

The flexibility of runtime fields comes at a cost: query latency. Because the computation happens during every query execution, runtime field lookups are slower than indexed fields. For high-throughput systems or latency-sensitive applications, this matters.

When Elasticsearch evaluates a runtime field, it must:

• Execute the Painless script for every matching document
• Evaluate the logic (math, string operations, conditionals)
• Generate the output value
• Apply any filters or aggregations to that value

For a small result set or infrequent queries, this overhead is negligible. For millions of documents scanned per second across a high-traffic endpoint, the cumulative cost becomes significant.

Let's benchmark a concrete example. Suppose you have a runtime field computing a discount tier based on order_amount, and you query 10 million documents with a filter on that tier. The query might take 2–5 seconds because the script must evaluate for each document. An equivalent indexed field would return in 200–500 milliseconds because the tier is already computed and stored.

This doesn't mean runtime fields are bad for production—it means they're best suited to:

• Low-frequency queries (analytics, batch jobs, reporting)
• Small result sets or targeted filters that eliminate most documents
• Temporary fields you're actively evaluating
• Backoffice operations where latency tolerance is higher

For customer-facing search, real-time dashboards, or high-frequency aggregations, indexed fields are the right choice once the logic is proven stable.

Index them later if used often

A common pattern we see in production is: start with a runtime field, validate it works, then graduate it to an indexed field once adoption is clear.

Here's how that workflow typically goes:

1. Week 1: Product team requests a new segmentation field. You define it as a runtime field to unblock analysis quickly.
2. Week 2–3: Dashboards and reports are built on top of it. Stakeholders confirm it's valuable. Query performance is acceptable because queries are run overnight or on-demand.
3. Week 4: A new dashboard goes live that queries this field every minute for 5 million documents. Latency spikes. You decide to index it.
4. Week 5: You reindex the data with the field now computed at ingest time. Queries drop from 3 seconds to 300 milliseconds. You remove the runtime field definition from the mapping.

To transition a runtime field to an indexed field, you typically:

1. Create a new index with the field in the regular mapping:

{
  "mappings": {
    "properties": {
      "line_total": {
        "type": "double"
      }
    }
  }
}

2. Reindex with the computation in an ingest pipeline:

{
  "source": {
    "index": "orders-old"
  },
  "dest": {
    "index": "orders-new"
  },
  "script": {
    "source": "ctx._source.line_total = ctx._source.unit_price * ctx._source.quantity"
  }
}

3. Once reindexing completes and validation passes, update aliases to point to the new index and retire the old one.

This staged approach is safer than committing expensive indexed storage upfront. You avoid the sunk cost of indexing fields that turn out to be rarely used, and you validate demand before investing in reindexing operations.

Common mistakes

We've seen several patterns that cause problems:

• Expensive scripts in high-QPS runtime fields: Complex string transformations or nested loops in Painless scripts compound under load. Profile your scripts and test query latency before deploying to production dashboards.
• Referencing missing fields without guards: If a runtime script calls doc['optional_field'].value and that field doesn't exist in a document, the query fails. Use doc['optional_field'].size() == 0 checks or params._source for safety.
• Aggregating on runtime fields at scale: Aggregating millions of documents by a runtime field computed on-the-fly is slow. If you need frequent aggregations, index the field first.
• Forgetting runtime fields in monitoring: Runtime fields consume CPU during queries but don't show up as storage cost. Monitor query latency and script execution time to catch performance regressions.
• Using runtime fields for security filtering: Never rely on a runtime field script as your only security control. A runtime field is transparent to the query and can be inspected or bypassed. Use Elasticsearch's document-level security instead.

Key takeaways

• Runtime fields let you compute derived values at query time without reindexing, making them ideal for rapid exploration and prototyping.
• They're slower than indexed fields because Painless scripts execute during every query, so reserve them for low-frequency queries or small result sets.
• Use runtime fields to validate new metrics or logic safely, then graduate them to indexed fields once you confirm they're frequently used and worth the storage cost.
• Always profile runtime field queries under realistic load and guard against missing fields and expensive script operations.
• Monitor script execution time as part of your query performance dashboard—runtime field cost is real even if storage cost isn't.