Search analytics: Elasticsearch best practices explained

Search analytics: Elasticsearch best practices explained

Most teams build search and then move on, never looking back at what users actually search for. The real competitive edge comes after launch, when you instrument your search layer to understand user intent and fix the gaps in your relevance. In this article, we'll walk through the practical steps to implement search analytics in Elasticsearch—logging queries, identifying failures, and turning that data into better search results.

Why search analytics matter

Search is a conversation between your users and your data. When someone types a query that returns zero results, or clicks on page three instead of the top result, that's valuable signal. We've all shipped a search feature that looked good in testing, only to discover users were searching for product names we'd never indexed, or common misspellings nobody had anticipated. Search analytics let you detect these patterns early and fix them systematically.

The goal isn't perfection on day one. The goal is fast feedback: understand what's breaking, what's working, and what your users actually want to find.

Log queries and clicks

Start by capturing the raw material: what users search for and what they click on. We typically log three events at minimum:

• Query event: the search string, timestamp, user ID, filters applied, and number of results returned
• Click event: which result was clicked, at what position, after how long
• Impression event: which results were shown (optional, but useful for ranking evaluation)

Store these in a separate index or even a dedicated logging system. Don't mix analytics events with your main search index—it'll bloat your shards and slow down queries.

Here's a realistic logging structure for a ride-hailing or e-commerce platform:

{
  "event_type": "search_query",
  "timestamp": "2025-01-15T14:32:10Z",
  "user_id": "user_123",
  "session_id": "sess_456",
  "query": "leather jacket",
  "query_normalized": "leather jacket",
  "filters": {
    "price_min": 50,
    "price_max": 200,
    "size": ["M", "L"]
  },
  "results_count": 127,
  "response_time_ms": 45,
  "source": "web"
}

And the corresponding click event:

{
  "event_type": "search_click",
  "timestamp": "2025-01-15T14:32:15Z",
  "user_id": "user_123",
  "session_id": "sess_456",
  "query": "leather jacket",
  "product_id": "prod_789",
  "rank": 3,
  "time_to_click_ms": 5200,
  "dwell_time_ms": 0
}

Normalize the query string consistently: lowercase, trim whitespace, remove special characters. This makes aggregation cleaner and patterns more visible. Use the same normalization in your search application and analytics pipeline—misalignment here creates subtle bugs that waste days later.

Find zero-result searches

A zero-result search is a scream for help. It means a user looked for something you don't have, your index structure is wrong, or your analyzer is too strict. Start by querying your analytics index for all searches with results_count equal to zero, grouped by the normalized query string.

{
  "size": 0,
  "query": {
    "match": {
      "event_type": "search_query"
    }
  },
  "aggs": {
    "zero_results": {
      "filter": {
        "term": {
          "results_count": 0
        }
      },
      "aggs": {
        "top_queries": {
          "terms": {
            "field": "query_normalized",
            "size": 20,
            "order": {
              "_count": "desc"
            }
          }
        }
      }
    }
  }
}

Run this daily. Look for patterns. If you see "mens shoes" returning zero results fifty times a week, that's data quality work waiting to happen. If it's rare one-off queries like "purple unicorn backpack," you might just ignore it. But if you see the same zero-result query ten times, investigate.

Common causes we've found in production:

• Typos and misspellings: "iphone 15 pro" when you've indexed "iPhone 15 Pro"
• Synonym gaps: users searching "TV" when you've indexed "television"
• Analyzer mismatch: stop words filtered out, or stemming too aggressive
• Data not indexed: the product exists in your database but wasn't passed to Elasticsearch
• Field mapping issues: text field analyzed, but you're running a term query

When we discovered our travel platform was getting zero results for "flights NYC to LAX," we found the problem: we'd indexed city names but not the airport codes. One synonym mapping fixed hundreds of monthly failures.

Add synonyms for common misses

Once you've identified the gaps, synonyms are your fastest fix. They're faster than reindexing, they work retroactively on existing data, and they're easy to tweak based on feedback.

Create a synonym filter in your index settings:

{
  "settings": {
    "analysis": {
      "filter": {
        "synonyms_filter": {
          "type": "synonym",
          "synonyms": [
            "tv, television",
            "iphone, apple phone",
            "usd => dollar, dollars, usd",
            "nyc, new york city",
            "jkt => jacket",
            "mens, men's, men",
            "ac, air conditioning, air conditioner"
          ]
        }
      },
      "analyzer": {
        "search_analyzer": {
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "stop",
            "synonyms_filter"
          ]
        }
      }
    }
  }
}

A few practical notes: synonyms are directional. tv => television means "tv expands to television," but tv, television means both expand to both. Use the arrow form when you want to normalize to a canonical term. Test synonyms before shipping—they can have unintended consequences if you're not careful.

In a SaaS CRM we worked on, we discovered users searching for "contact" when they meant "lead." Adding that synonym cut zero-result searches by twelve percent in the first week. But we also got noise—some users actually wanted to search the contacts table. So we refined it to context-specific synonyms, applied at query time only for certain fields.

The best approach: start with query-time synonyms applied to a dedicated analyzer, so you can tune them without reindexing. Once you're confident, bake high-impact synonyms into your index mapping.

Track click-through rate

Click-through rate (CTR) is your signal that relevance is working. High CTR on top results means your ranking is good. Low CTR with high result counts means either you have too many results or the first page isn't relevant.

Calculate CTR by position:

{
  "size": 0,
  "query": {
    "range": {
      "timestamp": {
        "gte": "now-7d"
      }
    }
  },
  "aggs": {
    "queries": {
      "filter": {
        "term": {
          "event_type": "search_query"
        }
      },
      "aggs": {
        "total_queries": {
          "cardinality": {
            "field": "session_id"
          }
        }
      }
    },
    "clicks": {
      "filter": {
        "term": {
          "event_type": "search_click"
        }
      },
      "aggs": {
        "by_rank": {
          "terms": {
            "field": "rank",
            "size": 10
          }
        }
      }
    }
  }
}

Once you have raw counts, calculate CTR for each position: clicks at rank 1 divided by total queries, clicks at rank 2 divided by total queries, etc. Healthy search typically shows:

• Position 1: 40–60% CTR
• Position 2: 15–25% CTR
• Position 3–5: 5–15% CTR
• Position 6+: under 5% CTR

If your position 1 CTR is below 30%, your top results aren't relevant. If it's above 70%, you might be too narrow. Drill deeper: which queries are underperforming? Are there categories or product types where ranking is consistently off?

We found in a travel search system that CTR for flight searches was strong (position 1 at 55%), but hotel searches were weak (position 1 at 18%). The difference: flights have clear differentiation (price, time), but our hotel search wasn't surfacing nearby properties first. We added geolocation boosting to the query, and CTR climbed to 42% within days.

Common mistakes

Not normalizing queries consistently: If your application lowercases queries but your analytics pipeline doesn't, you'll count "iPhone" and "iphone" as separate queries and miss the pattern entirely.

Logging too much data: Every event with every detail can balloon your index size. Be selective. Log user ID, session ID, timestamp, the query, and result count. You can add rich context later if needed—don't log it all upfront.

Ignoring rare queries: One-off weird searches are noise, but if the same weird query appears five times in a month, it's signal. Set a threshold (e.g., at least 5 occurrences in 30 days) before acting on it.

Applying synonyms too broadly: "Bat" is an animal and a sporting tool. Over-synonymizing creates false positives. Use context: if you're in sports retail, "bat" doesn't need to expand to "animal." Test in staging.

Treating all queries equally: A query that happens once is different from a query that happens fifty times. Weight your analysis by frequency and impact.

Key takeaways

• Log queries and clicks to a separate analytics index, capturing normalized query string, result count, user context, and click position.
• Run weekly aggregations for zero-result searches; the top recurring failures are your highest-ROI fixes.
• Use synonym mappings to close gaps between user language and indexed data, starting with query-time synonyms for fast iteration.
• Track click-through rate by position; position 1 CTR below 30% signals relevance problems worth investigating.
• Normalize consistently across application and analytics pipelines to avoid counting the same query twice.
• Act on patterns, not outliers; threshold analytics to focus on high-frequency issues.