Doc values and field data: practical tips and trade-offs

Introduction

By day 99, you've seen how Elasticsearch powers search, filtering, and analytics at scale. But sorting results by a field or running an aggregation on millions of documents can become a bottleneck if you don't understand how Elasticsearch retrieves and stores data for these operations. Doc values and field data: practical tips and trade-offs are the two mechanisms that make sorting and aggregations possible—and choosing between them, or configuring them correctly, can mean the difference between a responsive system and one that crashes under load.

This final piece in our series covers how these features work, when to use each, and the production gotchas we've learned the hard way.

Doc values are columnar on disk

Elasticsearch uses doc values for efficient sorting and aggregations on numeric, keyword, and date fields. Instead of storing data row-by-row (inverted index), doc values store it column-by-row on disk: all values for field status are clustered together, all values for price are clustered together. This columnar layout is much faster for aggregations and sorting because you only load the columns you need.

When you define a mapping, doc values are enabled by default for keyword and numeric types:

{
  "mappings": {
    "properties": {
      "status": {
        "type": "keyword",
        "doc_values": true
      },
      "price": {
        "type": "integer",
        "doc_values": true
      },
      "created_at": {
        "type": "date",
        "doc_values": true
      }
    }
  }
}

In a real e-commerce scenario, imagine you're running a query to find the average price of products by category. Elasticsearch loads the doc values for the price field (one column) and the category field (another column), then buckets by category and computes the average. The disk I/O is minimal because only two columns are touched.

Doc values live on disk, not in memory. A small portion is cached in memory via the field data cache (for frequently accessed fields), but the bulk stays on disk and is memory-efficient. For fields with high cardinality (many unique values), doc values still perform well because they're optimized for columnar access patterns.

Enabled by default for keyword and numbers

Elasticsearch automatically enables doc values for the following field types:

• keyword
• numeric (long, integer, short, byte, double, float, half_float, scaled_float)
• date
• boolean
• geo_point
• geo_shape
• ip

You rarely need to enable doc values manually for these types. However, there's one scenario where you might disable them: if you have a field that's only used for full-text search and never for sorting or aggregations, you can save disk space by setting "doc_values": false.

{
  "mappings": {
    "properties": {
      "description": {
        "type": "text",
        "doc_values": false
      }
    }
  }
}

In practice, we've rarely disabled doc values on production systems because the disk overhead is modest (typically 5–15% of index size), and the safety margin is worth it. If a stakeholder asks for a new metric based on that field later, you won't need to reindex.

For numeric and keyword fields, doc values are always a good idea. They're compressed on disk and require minimal memory overhead.

Field data on text is memory heavy

Text fields do not have doc values enabled by default. When you try to sort or aggregate on a text field, Elasticsearch falls back to field data: it loads the entire inverted index into memory and constructs a columnar structure on the fly. This is slow and expensive.

Here's a concrete example. You have a product index with a title field (type: text) and you attempt to aggregate products by title:

{
  "aggs": {
    "titles": {
      "terms": {
        "field": "title",
        "size": 10
      }
    }
  }
}

Elasticsearch will load field data for the title field into memory. If your title field has high cardinality (many unique values), this can consume gigabytes of heap space. In a high-traffic e-commerce environment, concurrent aggregations on different fields can quickly exhaust heap memory and trigger garbage collection pauses or out-of-memory errors.

We learned this the hard way during a Black Friday sale. A report query tried to aggregate orders by a text field (buyer notes), and the heap spiked to 95%. Subsequent queries queued up, and the cluster became unresponsive.

The solution is to use a keyword field instead:

{
  "mappings": {
    "properties": {
      "title": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword"
          }
        }
      }
    }
  }
}

Now you can aggregate on title.keyword without triggering field data:

{
  "aggs": {
    "titles": {
      "terms": {
        "field": "title.keyword",
        "size": 10
      }
    }
  }
}

The doc values for the keyword subfield are efficient and memory-light. This pattern—text field with a keyword subfield—is standard in production systems.

Aggregate on keyword fields

Whenever you need to sort or aggregate, use a keyword field (or numeric, date, boolean). Keyword fields are designed for exact matching and faceting, with doc values enabled by default. They're fast and predictable.

In a SaaS CRM environment, you might have:

{
  "mappings": {
    "properties": {
      "company_name": {
        "type": "text",
        "fields": {
          "keyword": {
            "type": "keyword",
            "ignore_above": 256
          }
        }
      },
      "industry": {
        "type": "keyword"
      },
      "annual_revenue": {
        "type": "long"
      },
      "created_date": {
        "type": "date"
      }
    }
  }
}

Aggregations work smoothly on industry (keyword), annual_revenue (long), and created_date (date). If you want to aggregate by company name, use company_name.keyword.

The ignore_above parameter on the keyword subfield is important: it tells Elasticsearch to ignore values longer than 256 characters. This prevents doc values from ballooning for accidentally long strings. Adjust this based on your domain (255 is a safe default for most use cases).

Aggregations are fast because Elasticsearch iterates through the doc values columns and buckets them efficiently. With text field data, that same operation would load gigabytes into memory.

Common mistakes

We've seen several patterns cause production pain:

• Aggregating on text fields directly. Always use a keyword subfield. If you inherit a schema without one, reindex with the keyword field added. The reindex can happen in parallel without downtime if you use index aliases.
• Disabling doc values on fields you later want to sort. Once disabled, you must reindex to enable them. Doc values cost little; the safety net is worth it.
• Using text for fields that are only exact-match. Enum-like fields (status, priority, country) should be keyword from the start. This saves memory and makes queries faster.
• Not monitoring field data cache eviction. If field data is filling up and being evicted, you're in trouble. Monitor indices.fielddata.memory.percent and set limits appropriately. If eviction is high, refactor your aggregations to use keyword fields instead.
• Assuming doc values are free. They do consume disk space and slightly increase indexing time. For append-only logs or very high-cardinality fields you'll never aggregate, disabling doc values is defensible. But the default is right for 95% of cases.

Key takeaways

• Doc values are columnar on-disk structures enabled by default for keyword, numeric, date, and boolean fields. They make sorting and aggregations fast and memory-efficient.
• Text fields do not have doc values by default. Aggregating on a text field loads field data into memory, which can exhaust heap and cause cluster instability.
• Always add a keyword subfield to text fields that you might want to aggregate or sort by. Use the fields parameter in your mapping.
• Set reasonable ignore_above values (256 or your domain's max) on keyword fields to prevent bloat.
• Monitor field data cache usage. High eviction rates mean you should refactor to use keyword fields instead.
• Doc values are compressed and efficient; disabling them is rarely necessary. The safe default is to keep them enabled.
• In production, the cost of a reindex to add a missing keyword field is outweighed by the memory and stability gains.