Cardinality aggregation: a practical Elasticsearch guide

Introduction

When you're running a high-traffic e-commerce platform or SaaS product, understanding how many unique visitors hit your site each day or how many distinct products users viewed is critical. Elasticsearch's cardinality aggregation gives you that answer fast, without consuming gigabytes of memory or timing out your analytics queries.

In this guide on cardinality aggregation: a practical Elasticsearch guide, we'll walk through how Elasticsearch leverages probabilistic data structures to count unique values at scale, when to trust its estimates, and how to tune precision for your specific use case.

How cardinality aggregation works with HyperLogLog

Cardinality aggregation in Elasticsearch relies on the HyperLogLog algorithm, a probabilistic method for counting distinct elements in a dataset. Unlike a naive approach where you'd store every unique value in memory, HyperLogLog maintains a small, fixed-size data structure and estimates cardinality with remarkably high accuracy.

The algorithm works by hashing each incoming value and analyzing the bit patterns of those hashes. It divides the hash space into buckets and tracks the longest run of leading zeros in each bucket. From these patterns, it mathematically estimates how many unique values must exist to produce that distribution of zeros.

Here's why this matters in production: if you're tracking unique user IDs across a billion events, HyperLogLog uses roughly 2 KB of memory per aggregation regardless of how many unique values you actually have. A traditional set-based approach would require storing every ID you've seen, potentially consuming gigabytes.

Let's see a basic cardinality query:

{
  "aggs": {
    "unique_users": {
      "cardinality": {
        "field": "user_id"
      }
    }
  }
}

When Elasticsearch executes this, it streams through all matching documents, hashes each user_id value, updates the HyperLogLog sketch, and returns an estimate. For most datasets, the estimate is within 5% of the true count.

Memory efficiency and speed benefits

The memory footprint of cardinality aggregation is one of its biggest selling points. By default, Elasticsearch allocates about 3 KB per cardinality aggregation, regardless of your data volume. This means you can run hundreds or thousands of cardinality aggregations on the same query without worrying about out-of-memory errors.

In a travel booking system, imagine you want to report on unique destination countries searched, unique hotels viewed, and unique users per region—all in one Elasticsearch query. With traditional approaches, building three separate counts could require materializing three large sets. Cardinality aggregations handle all three simultaneously with minimal overhead.

Speed is equally important. Because HyperLogLog doesn't require sorting or storing individual values, cardinality aggregations typically complete in milliseconds even on multi-billion document indices. Compare this to a terms aggregation with high cardinality, which must fetch and sort potentially millions of buckets.

In production, we've observed cardinality queries completing in 50–200 ms on a 50-shard cluster with 2 billion documents, whereas naive counting approaches would timeout or exhaust memory. The tradeoff is accuracy; you get an estimate, not an exact count. For most analytics and reporting use cases, a 2–5% margin of error is acceptable.

Controlling precision with precision_threshold

Elasticsearch exposes one key parameter to control the accuracy–memory tradeoff: precision_threshold. This setting tells Elasticsearch how many unique values you expect to see. If you set it correctly, HyperLogLog will use extra memory for small cardinalities and switch to estimation mode only when you exceed that threshold.

The default precision_threshold is 3000. This means Elasticsearch will maintain exact counts for the first 3000 unique values, then switch to probabilistic estimation. If you know your cardinality is always under 3000 (say, a list of product categories), you can leave it at default or increase it. If you're counting unique IP addresses globally and expect millions, you can lower it to save memory.

Here's an example tuning precision for a user behavior analytics pipeline:

{
  "aggs": {
    "unique_visitors": {
      "cardinality": {
        "field": "visitor_id",
        "precision_threshold": 10000
      }
    },
    "unique_products_viewed": {
      "cardinality": {
        "field": "product_id",
        "precision_threshold": 500
      }
    }
  }
}

In this query, we're saying "we'll have tens of thousands of unique visitors, so use estimation; but product IDs are limited to a few hundred, so maintain exactness." Elasticsearch will allocate memory accordingly.

A practical note: increasing precision_threshold from 3000 to 40000 multiplies memory usage by roughly 8–10x. On a 100-shard cluster, that can add meaningful overhead. Test with your actual data before deploying high precision_threshold values to production.

Real-world scenario: tracking unique visitors

Let's build a complete example: a ride-hailing platform wants to report daily unique riders and unique drivers. They ingest events into Elasticsearch with a timestamp, a user_id, and a user_type (rider or driver).

{
  "size": 0,
  "query": {
    "range": {
      "timestamp": {
        "gte": "2024-01-15T00:00:00Z",
        "lt": "2024-01-16T00:00:00Z"
      }
    }
  },
  "aggs": {
    "riders": {
      "filter": {
        "term": {
          "user_type": "rider"
        }
      },
      "aggs": {
        "unique_count": {
          "cardinality": {
            "field": "user_id",
            "precision_threshold": 50000
          }
        }
      }
    },
    "drivers": {
      "filter": {
        "term": {
          "user_type": "driver"
        }
      },
      "aggs": {
        "unique_count": {
          "cardinality": {
            "field": "user_id",
            "precision_threshold": 10000
          }
        }
      }
    }
  }
}

This query runs in a single pass, filtering by date, then computing separate cardinality estimates for riders and drivers. The response might be:

{
  "aggregations": {
    "riders": {
      "unique_count": {
        "value": 487532
      }
    },
    "drivers": {
      "unique_count": {
        "value": 12847
      }
    }
  }
}

In under 100 ms, you have your daily active user counts. These estimates are reliable for dashboards, alerting, and business intelligence. The precision_threshold values reflect realistic cardinality: you expect many riders, fewer drivers.

Common mistakes

Setting precision_threshold too high. Developers often think "more precision is always better" and set it to 1 million. This balloons memory usage and slows aggregations with no practical benefit if your true cardinality is only in the thousands. Start at the default 3000 and increase only after measuring real cardinality in your data.

Confusing cardinality with distinct counts in subgroups. If you nest a cardinality aggregation inside a terms aggregation, each bucket maintains its own HyperLogLog sketch. The sum of all buckets' cardinality values does not equal a top-level cardinality aggregation. This is correct behavior but catches many teams off guard.

Using cardinality on high-precision fields without normalization. If you're counting unique email addresses but some are stored as "John@Example.com" and others as "john@example.com", your cardinality will be artificially high. Normalize at index time or use a preprocessor in your ingestion pipeline.

Ignoring that cardinality is an estimate. For compliance or financial reporting where you need exact counts, cardinality aggregation is not appropriate. Use it for analytics, feature flagging decisions, and performance monitoring where a 3–5% error margin is acceptable.

Not testing precision_threshold with production cardinality. A development environment with 1000 unique users doesn't tell you how precision_threshold will behave with 10 million. Always run benchmarks on a production data sample before deploying changes.

Key takeaways

• Cardinality aggregation uses HyperLogLog, a probabilistic algorithm that estimates unique counts with fixed, tiny memory overhead (typically 2–3 KB).
• It's ideal for high-cardinality fields like user IDs, IP addresses, or session tokens, especially when you need fast aggregations across billions of documents.
• The precision_threshold parameter controls the accuracy–memory tradeoff; set it to match your expected cardinality distribution.
• Cardinality estimates are typically within 5% of true counts and are suitable for analytics, dashboards, and monitoring but not for exact financial or compliance reporting.
• Nest cardinality inside filters and terms aggregations to slice counts by dimension; each aggregation maintains its own HyperLogLog sketch.
• Test precision_threshold values with realistic production data and cardinality levels before deploying to avoid memory surprises.