Testing search relevance: a hands-on guide for engineers

Testing search relevance: a hands-on guide for engineers

Search relevance feels like magic until it doesn't. You tune a query, release it to production, and suddenly users complain that results feel worse, not better. The gap between what we think works and what actually works is where most search projects stumble. In this guide, we'll walk through the practical methods we use to measure search quality before and after changes, turning gut feelings into data-driven decisions.

Why measuring relevance matters before you tune

Relevance is subjective, but its impact is concrete: poor search directly kills conversions and engagement. Yet many teams skip the measurement phase entirely, jumping straight to configuration changes. We've seen this pattern repeat across e-commerce platforms, SaaS dashboards, and internal knowledge bases. Without a baseline, you can't prove your tuning actually helped. Worse, you might optimize for one metric while breaking another.

Testing search relevance isn't optional overhead—it's the only way to ship search improvements with confidence. The good news is that Elasticsearch gives us the tools to do this systematically.

Build a set of queries and expected results

Start with what your users actually search for. Pull your top 100-200 queries from the past month. For an e-commerce site, this might include brand names, product types, and common misspellings. For a travel platform, it's destinations, date ranges, and amenities. For internal tools, it's feature names and common documentation searches.

For each query, define the ideal result set. This is judgment work, not automation. Sit down with a product manager, domain expert, or power user and answer: what are the top 5-10 results we'd want to show for this query? Document the expected document IDs or unique identifiers.

Here's what a simple test set looks like:

{
  "test_queries": [
    {
      "query": "wireless headphones under $100",
      "expected_results": [
        {
          "id": "prod_12847",
          "product_name": "BudsPro Lite",
          "rank": 1
        },
        {
          "id": "prod_8392",
          "product_name": "SoundGear Basic",
          "rank": 2
        },
        {
          "id": "prod_15603",
          "product_name": "QuietMax 3",
          "rank": 3
        }
      ]
    },
    {
      "query": "laptop 16 inch gaming",
      "expected_results": [
        {
          "id": "prod_22451",
          "product_name": "GamerMax Pro 2024",
          "rank": 1
        },
        {
          "id": "prod_21899",
          "product_name": "PowerBook Elite",
          "rank": 2
        }
      ]
    }
  ]
}

Start with 20-30 queries if you're new to this. You can expand later. The goal is coverage of your most important search patterns, not exhaustive completeness.

Rank Evaluation API

Elasticsearch's Rank Evaluation API is built exactly for this. It runs your test queries against your current index and compares the actual results against your expected results, giving you numerical scores. This is how we move from "this feels better" to "precision improved 12%."

The API supports several ranking metrics. The ones we use most often are:

• Precision at K: Of the top K results, how many are relevant? Precision@10 means: of the top 10 results, how many did we expect?
• Recall at K: Of all the relevant results, how many appear in the top K?
• Discounted Cumulative Gain (DCG): Accounts for ranking order. A relevant result in position 1 is worth more than position 5.
• Mean Reciprocal Rank (MRR): Where is the first relevant result? Useful for queries where one correct answer matters most.

Here's a practical example. Let's say you have a test set and want to measure precision@5 and DCG@10:

{
  "requests": [
    {
      "id": "query_1",
      "request": {
        "query": {
          "multi_match": {
            "query": "wireless headphones under $100",
            "fields": ["name^2", "description", "tags"]
          }
        }
      },
      "ratings": [
        { "rating": 1, "_index": "products", "_id": "prod_12847" },
        { "rating": 1, "_index": "products", "_id": "prod_8392" },
        { "rating": 1, "_index": "products", "_id": "prod_15603" },
        { "rating": 0, "_index": "products", "_id": "prod_9999" }
      ]
    },
    {
      "id": "query_2",
      "request": {
        "query": {
          "multi_match": {
            "query": "laptop 16 inch gaming",
            "fields": ["name^2", "description", "tags"]
          }
        }
      },
      "ratings": [
        { "rating": 1, "_index": "products", "_id": "prod_22451" },
        { "rating": 1, "_index": "products", "_id": "prod_21899" },
        { "rating": 0, "_index": "products", "_id": "prod_11111" }
      ]
    }
  ],
  "metric": {
    "precision": {
      "k": 5
    },
    "dcg": {
      "k": 10
    }
  }
}

In the ratings array, 1 means relevant, 0 means not relevant. When you run this request against your Elasticsearch cluster, you get back scores for each query and an overall average:

{
  "metric_score": 0.674,
  "details": {
    "query_1": {
      "metric_score": 0.8,
      "precision": 0.8,
      "dcg": 0.92
    },
    "query_2": {
      "metric_score": 0.55,
      "precision": 0.4,
      "dcg": 0.67
    }
  }
}

That metric_score becomes your baseline. Every time you change your query logic, boosting, or analyzers, you run this evaluation again and compare the new score against the baseline.

Compare changes against baseline

Once you have a baseline score, changes become measurable. Let's say you're considering adding synonym expansion or adjusting field boosts. Here's the workflow we follow:

1. Document your current query configuration. Save the exact query DSL you're evaluating against.
2. Make one change. Adjust one thing at a time—add synonyms, boost brand field, or change analyzer settings. Not ten things at once.
3. Re-run the rank evaluation against the same test set.
4. Compare the new score to the baseline. Did it improve? By how much?
5. If it improved, keep the change and use the new score as your baseline. If it regressed, revert and try something different.

We keep a simple spreadsheet tracking this:

Each row is a hypothesis you tested. Over time, you stack improvements and they compound. A 2-3% gain per change doesn't sound like much, but three approved changes mean 6-9% total improvement in your relevance metric.

Review with real users

Metrics are powerful but incomplete. A numerical score can hide real problems. Maybe your MRR improved but power users hate the new order. Maybe precision went up but you're showing fewer diverse results. This is why we always validate metric improvements with actual users before calling a change done.

There are a few ways to do this:

A/B testing in production: Route a percentage of traffic to the new query configuration and measure click-through rate, dwell time, or conversion. This is the gold standard but requires instrumentation and traffic volume.

Manual review by power users: Show your test queries and the old vs. new result sets to domain experts. Ask them which feels better and why. This takes a few hours but catches issues metrics miss. We usually do this before A/B testing.

Usability testing: Have actual users perform search tasks with both configurations and observe where they succeed or get stuck. Video record these sessions so the team can watch where search failed them.

In one SaaS project, our metrics said the new relevance was better, but users in testing said results felt less organized. Turned out we'd hurt category-based browsing even though individual query precision improved. We adjusted the change and re-tested both metrics and users. Metrics alone would have shipped a regression.

Common mistakes

• Test set too small: 5-10 queries won't catch edge cases. Aim for at least 20-30, distributed across your major search patterns. You'll find surprising variations.
• Ratings done by one person: One person's idea of relevance is biased. Get at least two raters per query, especially for subjective domains like content recommendations. When they disagree, discuss why.
• Changing multiple things at once: If you boost fields, add synonyms, and adjust the analyzer all in one change, and metrics improve, which change helped? You won't know. You'll also make debugging harder. Change one thing, test, learn, then move to the next.
• Overfitting to test queries: Your test set is representative, not exhaustive. A change that improves your 30 test queries might break results for queries similar to those 30. Run your changes against a separate validation set if possible, or at least ask teammates to try them.
• Ignoring latency: A change that improves relevance by 5% but doubles query latency is a bad trade. Always measure both relevance and performance together.
• Forgetting to re-test after indexing changes: If you update your analyzer or add new fields, re-run your evaluation against the updated index. Metrics can shift unexpectedly.

Key takeaways

• Build a test set of 20-30 queries and expected results before you tune. This becomes your relevance baseline.
• Use Elasticsearch's Rank Evaluation API to measure precision, recall, DCG, or MRR. Pick metrics that matter for your use case.
• Make one change at a time, re-evaluate, compare to baseline. Stack improvements incrementally.
• Always validate metric improvements with real users before shipping. Metrics are guide, not gospel.
• Track your changes and scores over time. You'll spot patterns and build confidence faster.
• Relevance tuning is iterative. You won't get it perfect on the first try, and that's fine. The process beats the destination.