Percolator queries in Elasticsearch & Search Engineering

The reverse search problem

Most of us think of search in one direction: a user enters a query, and we return matching documents. But what if we flipped that on its head? What if documents arrived continuously, and we needed to instantly match them against thousands of stored queries? That's the core idea behind percolator queries in Elasticsearch & Search Engineering—and it solves real business problems that traditional indexing simply cannot handle efficiently.

Whether you're building price drop alerts, fraud detection rules, or notification systems, the percolator gives us a way to store queries as documents and match incoming data against them at scale. Let's explore how this reverse search pattern works and when to reach for it in production.

Store queries, match incoming documents

In a standard search flow, documents live in the index and queries are ephemeral—they arrive, execute, and disappear. The percolator inverts this: queries become first-class citizens stored as indexed documents. When a new document arrives, instead of running it through a query parser, we ask Elasticsearch: "Which stored queries does this document match?"

Under the hood, Elasticsearch converts each percolator query into an internal representation and stores it. When you percolate a document, the engine feeds it through all stored queries simultaneously, returning the IDs of queries that matched. This is why it's sometimes called "reverse search"—the query engine now runs from document to queries, not queries across documents.

Here's a minimal example. First, we define a percolator query to store:

PUT /products/_doc/price_alert_1?refresh
{
  "query": {
    "range": {
      "price": {
        "lte": 50
      }
    }
  }
}

Notice there's no special syntax yet—this is just storing a query document. To use it, we need to set up the index mapping correctly, which we'll cover next. When a new product document comes in with a price of $45, we percolate it:

GET /products/_search
{
  "query": {
    "percolate": {
      "field": "query",
      "document": {
        "price": 45,
        "name": "Widget"
      }
    }
  }
}

The response tells us which stored queries matched:

{
  "hits": {
    "hits": [
      {
        "_id": "price_alert_1",
        "fields": {
          "query": [
            {
              "query": "price_alert_1"
            }
          ]
        }
      }
    ]
  }
}

A single percolate request can match dozens or hundreds of stored queries in milliseconds, making it ideal for fan-out notification systems.

Saved search alerts

One of the most common applications is saved search alerts. Imagine a SaaS platform where users save complex search criteria—like "used cars under $20k, 2015 or newer, within 50 miles." Every time a new listing arrives, you need to notify all users whose saved searches match.

Without the percolator, you'd query each saved search independently against the new document, which becomes prohibitively slow at scale. With the percolator, all saved searches live in Elasticsearch as indexed queries, and a single percolate request tells you instantly which users to notify.

Let's build this properly. First, create the index with the right mapping:

PUT /saved_searches
{
  "mappings": {
    "properties": {
      "query": {
        "type": "percolator"
      },
      "user_id": {
        "type": "keyword"
      },
      "created_at": {
        "type": "date"
      }
    }
  }
}

Now store a user's saved search:

PUT /saved_searches/_doc/search_user_42?refresh
{
  "user_id": "user_42",
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "year": {"gte": 2015}
          }
        },
        {
          "range": {
            "price": {"lte": 20000}
          }
        },
        {
          "geo_distance": {
            "distance": "50km",
            "location": {
              "lat": 40.7128,
              "lon": -74.0060
            }
          }
        }
      ]
    }
  },
  "created_at": "2025-01-15T10:00:00Z"
}

When a new listing arrives, percolate it:

GET /saved_searches/_search
{
  "query": {
    "percolate": {
      "field": "query",
      "document": {
        "year": 2018,
        "price": 18500,
        "location": {
          "lat": 40.7128,
          "lon": -74.0060
        }
      }
    }
  }
}

The response includes the user_id field, letting you immediately dispatch a notification. This pattern scales from dozens to millions of saved searches without needing a message queue or background job to iterate through each one.

Price drop notifications

E-commerce and travel platforms rely heavily on price drop alerts. A user watches a flight or hotel, and wants to know the instant it drops below their target price. Using a traditional search architecture, you'd either poll periodically or maintain a separate alerting system. Both are inefficient.

The percolator lets you index price-watch queries once and match them as prices stream in. Here's a realistic example for an e-commerce platform:

PUT /price_watches
{
  "mappings": {
    "properties": {
      "query": {
        "type": "percolator"
      },
      "user_id": {
        "type": "keyword"
      },
      "product_category": {
        "type": "keyword"
      },
      "alert_threshold": {
        "type": "float"
      }
    }
  }
}

Store a watch:

PUT /price_watches/_doc/watch_user_99_sku_7734?refresh
{
  "user_id": "user_99",
  "product_category": "electronics",
  "alert_threshold": 299.99,
  "query": {
    "bool": {
      "must": [
        {
          "term": {
            "sku": "7734"
          }
        },
        {
          "range": {
            "current_price": {
              "lte": 299.99
            }
          }
        }
      ]
    }
  }
}

When the price updates, percolate it:

GET /price_watches/_search
{
  "query": {
    "percolate": {
      "field": "query",
      "document": {
        "sku": "7734",
        "current_price": 279.99,
        "timestamp": "2025-01-15T14:32:00Z"
      }
    }
  }
}

The percolator response tells you which watches matched. You can then enrich it with metadata from your stored watch documents and send notifications immediately. This approach avoids maintaining a separate event queue and makes alerting logic centralized and version-controlled in your mappings.

Scales to many stored queries

One of the percolator's greatest strengths is its ability to handle thousands or tens of thousands of stored queries without degradation. Each percolate request visits all candidate queries, but Elasticsearch optimizes this with internal indexing and query filtering.

However, scale comes with caveats. A few practical guidelines:

• Query complexity matters more than count. A percolator with 50,000 simple term queries performs better than 1,000 complex boolean queries with nested aggregations.
• Use filters judiciously. The percolate query accepts a filter parameter that can pre-filter stored queries before evaluation. This dramatically reduces the number of queries examined.
• Segment size affects latency. Frequent flushes create more segments to scan. Monitor segment count and optimize refresh intervals if percolate latency spikes.
• Memory and CPU are bottlenecks. Unlike document search, percolation is CPU-bound. If you have very high document throughput, you may need dedicated percolator nodes or a separate cluster.

Here's an example using a filter to reduce the query set before percolation:

GET /price_watches/_search
{
  "query": {
    "bool": {
      "must": [
        {
          "percolate": {
            "field": "query",
            "document": {
              "sku": "7734",
              "current_price": 279.99
            }
          }
        }
      ],
      "filter": [
        {
          "term": {
            "product_category": "electronics"
          }
        }
      ]
    }
  }
}

This filters the stored query set to only electronics before percolating, cutting the number of evaluations significantly.

Common mistakes

Forgetting the percolator mapping type. The most frequent mistake is trying to use percolator queries without explicitly declaring the field as type "percolator" in your mapping. Without it, Elasticsearch treats the query as a nested object and percolation fails silently or throws an error.

Percolating too much data. Sending massive nested documents into a percolate query is wasteful. Only include the fields that your stored queries actually reference. Extra fields add serialization overhead and serve no purpose.

Not filtering before percolation. If your stored queries are categorized (product type, region, user tier), always filter to the relevant subset before percolating. This can cut latency by 50–80% and is often overlooked.

Treating percolator like a cache. Some teams build alerting systems expecting percolator queries to update instantly when documents change. The percolator is stateless—it matches documents to queries at query time, not as a reactive system. If you need true reactive alerts, combine it with a message broker.

Ignoring shard allocation. Percolate requests are coordinated queries. If your percolator index is fragmented across many shards with an uneven query distribution, you'll see hotspots. Keep percolator shards balanced and consider a dedicated index per use-case if throughput is high.

Key takeaways

• Percolator queries reverse the search direction: store queries as documents, match incoming data against them instantly.
• Declare the percolator field type explicitly in your mapping; without it, the feature won't work.
• Use percolators for saved searches, price alerts, fraud detection rules, and any alert system where queries are long-lived and documents are ephemeral.
• Always filter the stored query set before percolation to reduce CPU load and improve latency.
• Scale expectations: tens of thousands of queries are feasible, but query complexity and document throughput determine real-world limits.
• Percolators are stateless query evaluators. Combine with event streams or message brokers for true reactive systems.
• Monitor segment count and refresh behavior; percolation latency is CPU-bound, not I/O-bound.