Cross-cluster search in Elasticsearch & Search Engineering

Hook

When your search infrastructure spans multiple data centers or regions, querying each cluster separately becomes a bottleneck. Cross-cluster search in Elasticsearch & Search Engineering lets you submit a single query from one coordinating cluster and fetch results from remote clusters transparently, reducing operational complexity and improving the developer experience across distributed teams.

Search remote clusters from one place

Cross-cluster search is the ability to issue a search request to one Elasticsearch cluster—the coordinating cluster—and have that request transparently executed against one or more remote clusters. Instead of opening separate connections and merging results in your application layer, Elasticsearch handles the network hop, aggregation, and ranking for you.

Imagine you operate search in three regions: US, EU, and APAC. Without cross-cluster search, a user in London searches only the EU cluster, missing potentially relevant products stored in the US cluster. With cross-cluster search enabled, a single request can fan out to all three regions and return unified results ranked by relevance, not by which cluster they came from.

The query syntax is straightforward. Instead of searching my-index, you search remote-cluster:my-index or even combine local and remote indexes:

{
  "query": {
    "multi_match": {
      "query": "wireless headphones",
      "fields": ["title^2", "description"]
    }
  }
}

When executed against us-cluster:products,eu-cluster:products,local:products, the coordinating node disseminates this query to all three clusters, collects hits, performs final ranking, and returns a unified result set. Your application code remains unchanged.

Useful for multi-region data

Multi-region deployments are common in modern infrastructure. You might replicate data across regions for compliance (GDPR requires EU data in EU), disaster recovery, or latency reduction. Cross-cluster search lets you leverage this geographic distribution without fragmenting your search logic.

Consider an e-commerce platform where product catalogs are region-specific but must be searchable across all markets:

• North America cluster: 10 million products, updated hourly.
• Europe cluster: 8 million products, regional variants.
• Asia-Pacific cluster: 5 million products, local sellers.

A customer traveling from the US to Europe should still see products from their home region mixed with local offerings. Instead of replicating 23 million documents to a single cluster—expensive and slow—you keep each region independent and use cross-cluster search to query all three simultaneously.

Another scenario: financial institutions running separate clusters per regulatory domain. A trading platform might have:

• US equities (SEC regulated)
• EU securities (ESMA regulated)
• Asia derivatives (local regulators)

Compliance rules prevent data co-location, but traders need a unified search across all domains. Cross-cluster search bridges this gap without violating isolation requirements.

Configure remote cluster connections

Setting up cross-cluster search requires configuring remote cluster endpoints on your coordinating cluster. This is typically done once, during infrastructure setup, but understanding the configuration is essential for troubleshooting and scaling.

In Elasticsearch 7.10+, you use the Cluster Settings API to register a remote cluster. For Elasticsearch 8.0+, you can also use Kibana's UI, but the API approach is more reliable for infrastructure-as-code practices.

Step 1: Define remote cluster settings

PUT _cluster/settings
{
  "persistent": {
    "cluster.remote.us-cluster": {
      "seeds": ["10.0.1.50:9300", "10.0.1.51:9300"]
    },
    "cluster.remote.eu-cluster": {
      "seeds": ["10.1.1.50:9300", "10.1.1.51:9300"]
    },
    "cluster.remote.apac-cluster": {
      "seeds": ["10.2.1.50:9300", "10.2.1.51:9300"]
    }
  }
}

The seeds parameter is a list of transport addresses (port 9300, not HTTP port 9200) for nodes in the remote cluster. Elasticsearch will discover other nodes automatically once it connects to any seed.

Step 2: Verify connection

GET _remote/info

This returns connection status for all registered remote clusters:

{
  "us-cluster": {
    "connected": true,
    "mode": "sniff",
    "seeds": ["10.0.1.50:9300", "10.0.1.51:9300"],
    "num_nodes_connected": 5,
    "initial_connect_timeout": "30s"
  },
  "eu-cluster": {
    "connected": true,
    "mode": "sniff",
    "seeds": ["10.1.1.50:9300", "10.1.1.51:9300"],
    "num_nodes_connected": 4,
    "initial_connect_timeout": "30s"
  }
}

If a cluster shows "connected": false, check network connectivity, firewall rules, and whether the transport port (9300) is open. A common mistake is using the HTTP port instead of the transport port.

Step 3: Query remote clusters

Once connected, query with the cluster name prefix:

GET us-cluster:products,eu-cluster:products/_search
{
  "size": 20,
  "query": {
    "bool": {
      "must": [
        {"match": {"title": "laptop"}},
        {"range": {"price": {"lte": 1500}}}
      ]
    }
  }
}

This queries the products index in both us-cluster and eu-cluster, returning the top 20 results from across both. You can also combine local and remote indexes: local:products,us-cluster:products.

Watch latency between regions

Cross-cluster search introduces network latency between regions. A query that takes 50ms locally might take 200–500ms when querying a remote cluster across continents. Understanding and optimizing this latency is critical for user experience.

Latency sources:

1. Network round trip: Coordinating cluster → remote cluster → back. At minimum, this is the speed of light through fiber, roughly 150ms for US to EU.
2. Query execution: The remote cluster still needs to search its shards, aggregate, and return results.
3. Shard allocation: Results must be gathered from multiple shards on the remote cluster, then forwarded back.
4. Ranking overhead: The coordinating cluster performs final ranking and sorting across results from multiple clusters, adding serialization and deserialization overhead.

Monitoring and observability

Use the search slow log and cluster metrics to identify latency bottlenecks:

GET us-cluster:products/_stats/search

Enable search slow logs on remote clusters to log queries taking longer than a threshold:

PUT us-cluster:_settings
{
  "index.search.slowlog.threshold.query.warn": "1s",
  "index.search.slowlog.threshold.fetch.warn": "500ms"
}

In your application, instrument cross-cluster queries with timing:

{
  "search_after": [1500, "product-2048"],
  "query": {...},
  "_name": "cross_region_query",
  "took": 287,
  "timed_out": false,
  "clusters": {
    "us-cluster": {"took": 150},
    "eu-cluster": {"took": 240},
    "coordination": {"took": 45}
  }
}

The took field in the response tells you total milliseconds. Break this down by cluster to identify which one is the slowest. If EU cluster queries are consistently slow, it might have higher load or older hardware.

Optimization strategies

Reduce latency by:

• Filtering before the network hop: Use query filtering to exclude remote queries that won't contribute results. For instance, if querying products by region, filter by region first to avoid sending queries to clusters unlikely to have matches.
• Asynchronous queries: For non-real-time features, use the async search API to submit queries that execute in the background while your app polls for results.
• Caching: Cache frequently-accessed cross-cluster query results in Redis or a similar store, with short TTLs to balance freshness and latency.
• Shard allocation awareness: Ensure remote cluster shards are allocated to nodes closest to your coordinating cluster, reducing intra-cluster latency.

Common mistakes

Using HTTP port instead of transport port: The seeds list must use port 9300 (transport/node communication), not 9200 (HTTP). A seed with 10.0.1.50:9200 will fail silently.

No network connectivity checks: Firewalls, VPCs, and NAT rules often block port 9300. Always test connectivity with nc -zv 10.0.1.50 9300 before assuming configuration is wrong.

Treating cross-cluster search as real-time replication: Cross-cluster search queries the remote cluster's current state; it's not a continuous sync. If you need strong consistency, consider a different approach.

Ignoring latency in SLAs: A 200ms cross-cluster query is slower than a 50ms local query. If your SLA is 100ms, cross-cluster search won't meet it without caching or prioritization strategies.

Querying too many clusters at once: Each additional remote cluster adds latency and complexity. Beyond 5–10 clusters, consider fan-out patterns or data aggregation at the application layer instead.

Not monitoring per-cluster metrics: If you treat cross-cluster latency as a single number, you'll miss slow clusters. Break down timings by cluster name to identify root causes.

Key takeaways

• Cross-cluster search allows a single query against multiple Elasticsearch clusters, simplifying distributed search architectures.
• Register remote clusters via the Cluster Settings API using transport addresses (port 9300), not HTTP ports.
• Multi-region deployments benefit from cross-cluster search by avoiding data replication while maintaining searchability across all regions.
• Latency is the primary challenge; monitor per-cluster query times and optimize with filtering, async APIs, and caching.
• Common pitfalls include wrong ports, missing network checks, and treating cross-cluster search as a replacement for replication.
• Keep the number of queried clusters reasonable (5–10 as a practical limit) to avoid cascading latency and operational overhead.