Learning to rank: Elasticsearch best practices explained

Learning to rank: Elasticsearch best practices explained

Most teams rely on BM25 and manually tuned queries to rank search results, but this approach plateaus quickly. Once you've optimized your query structure and field boosting, improving relevance requires something more sophisticated: machine learning. Learning to rank: Elasticsearch best practices explained shows us how to move beyond static ranking rules and let your actual user behavior drive better results.

Why static ranking isn't enough

Consider an e-commerce platform where you've tuned your Elasticsearch queries for months. You boost product title matches, adjust field weights, and use custom scoring. But you notice that users searching for "running shoes" click on mid-priced athletic brands, not the highest-priced premium options. Your query ranks by relevance score alone, missing this behavioral signal entirely.

Static ranking rules can't adapt to domain-specific patterns that only appear in your data. A travel site's query might need to balance recency, user ratings, and seasonal demand differently than a news site would. This is where learning to rank (LTR) becomes valuable. Instead of manually writing rules, you train a model on historical user interactions, and the model learns which ranking factors matter most for your specific use case.

Train models on click data

The foundation of effective LTR is clean, labeled training data. Click data is the most practical source: when a user searches for "laptop" and clicks on result 3 instead of result 1, that's a signal that result 3 was more relevant to that user's intent.

Collecting the right signals:

• Clicks: Which results users interact with, and at what position (position bias matters—users click higher results more often even if they're not best)
• Dwell time: How long users spend on a result page before returning to search
• Conversions: Which search sessions led to purchases, sign-ups, or completed actions
• Skips: When users skip over a result to click a lower-ranked one, that's strong evidence the skipped result was less relevant
• Rate and feedback: Explicit ratings or thumbs-up/thumbs-down signals if your UI supports them

A typical pipeline looks like this: log every search query and its result positions, track which results were clicked, note user context (device, location, previous behavior), and export this data periodically for model training. You'll want at least a few thousand labeled examples per query intent, though thousands of examples across your entire corpus work too.

In practice, you might track clicks like this in your application:

{
  "query": "running shoes",
  "user_id": "user_456",
  "session_id": "sess_789",
  "timestamp": "2024-01-15T14:32:00Z",
  "results": [
    {
      "doc_id": "product_1001",
      "position": 1,
      "clicked": false
    },
    {
      "doc_id": "product_2847",
      "position": 2,
      "clicked": true,
      "dwell_time_ms": 4200
    },
    {
      "doc_id": "product_1502",
      "position": 3,
      "clicked": false
    }
  ]
}

This log tells us that product_2847 was more relevant than products at positions 1 and 3 for this particular query. Across thousands of such events, patterns emerge about which document features matter most.

The LTR plugin: Elasticsearch and OpenSearch implementation

Both Elasticsearch and OpenSearch offer Learning to Rank plugins that handle model serving directly in the search engine. This approach keeps ranking logic close to the data and eliminates the latency of external ranking services.

Elasticsearch LTR (community-driven): The Elasticsearch Learning to Rank plugin lets you upload trained models and score documents using a custom query clause. Models are typically in LambdaMART, XGBoost, or similar formats, and the plugin applies them during query execution.

OpenSearch ML Plugin: OpenSearch includes native ML Commons, which supports model upload, versioning, and serving. You can upload your trained model once and reference it across multiple indices.

To use LTR in Elasticsearch, you'd typically store your model in the `.ltr` index and reference it in a query:

{
  "query": {
    "bool": {
      "must": [
        {
          "match": {
            "title": "running shoes"
          }
        }
      ],
      "should": [
        {
          "ltr": {
            "model": "my_ranking_model",
            "params": {
              "keywords": "running shoes",
              "user_tier": "premium"
            }
          }
        }
      ]
    }
  }
}

The LTR query clause computes features for each matching document and applies your trained model to generate a ranking score. Results are then sorted by this LTR score instead of (or in addition to) BM25 relevance.

Features from fields and behavior

A good LTR model needs meaningful input features. These come from two main sources: document fields and behavioral signals.

Field-based features:

• BM25 scores for title, description, tags
• Query-document term overlap (how many query terms appear in the document)
• Document freshness (days since last update)
• Document popularity (view count, link count)
• Static quality scores (editorial ratings, seller reputation)
• Price, category, brand—domain-specific attributes

Behavioral features:

• Click-through rate for this document given this query
• Average dwell time on documents of this type
• Conversion rate (purchase, signup) for this query-document pair
• User personalization (whether the user has purchased from this seller before)

In practice, you'd compute these features within Elasticsearch using painless scripting or externally and store them as denormalized fields. Here's an example of feature computation at indexing time:

{
  "mappings": {
    "properties": {
      "title": {
        "type": "text"
      },
      "description": {
        "type": "text"
      },
      "ltr_features": {
        "type": "nested",
        "properties": {
          "popularity_score": {
            "type": "rank_feature"
          },
          "freshness_days": {
            "type": "rank_feature"
          },
          "seller_rating": {
            "type": "rank_feature"
          }
        }
      }
    }
  }
}

Then, when training your model offline, you'd extract these features alongside your click labels:

A trained model learns that for "running shoes," popularity and freshness matter almost as much as raw BM25 score. The model might learn that a document with a lower BM25 score but very high popularity should rank higher. This is something you'd never tune manually without extensive A/B testing.

Needs good data first

The most common reason LTR projects fail is insufficient or biased training data. You can't train a model to rank for queries that have almost no clicks, and you can't improve ranking with data that reflects only current (possibly broken) ranking behavior.

Data quality checklist:

• Coverage: Do you have clicks for the query intents you care about most? A model trained only on high-volume queries won't help long-tail search.
• Bias correction: Clicks are biased toward highly ranked results. Use position-aware metrics like normalized discounted cumulative gain (NDCG) or consider re-ranking clicks with position bias corrections before training.
• Temporal relevance: If your data spans months or years, ranking preferences may have shifted. Retrain regularly or use recent data only.
• Negative examples: You need both clicked (positive) and non-clicked (negative) examples. If every result looks clicked, your model learns nothing.
• Diversity: Ensure your training set includes results the model should rank low, not just variations of good results.

Before training any model, audit your click logs. What percentage of your queries have at least one click? How many queries have 20+ clicks? If most queries have zero clicks, you don't have enough signal yet. Consider running controlled experiments first to generate better behavioral data, or start with domain expertise to manually label a smaller set of query-result pairs.

Common mistakes

Training on current ranking data: If your training set contains clicks from your current (mediocre) ranking system, your model will learn to replicate those rankings, not improve them. Aim for clicks from a more randomized or A/B-tested baseline.

Ignoring position bias: Users click results at position 2 more often than position 10, even if position 10 is better. Without adjusting for this, your model learns that position matters more than relevance. Use inverse propensity weighting or position-based bias models during training.

Too few features: Feeding your model only BM25 scores gives it almost nothing to learn. Include document metadata, freshness, popularity, and user signals. More features (up to a point) help the model discover ranking patterns.

Not retraining: Click behavior changes over time. User preferences shift, new products arrive, and old click data becomes stale. Set up automated retraining on a weekly or monthly cadence, and monitor model performance to detect degradation.

Over-engineering before validation: Don't spend weeks building the perfect feature pipeline before testing whether LTR helps at all. Start with simple features, train a basic model, and measure impact. If it works, invest in sophistication.

Key takeaways

• Learning to rank moves beyond static tuning by letting user behavior drive ranking decisions through machine learning models.
• Click data, dwell time, and conversion signals form the foundation of training data; collect signals systematically and monitor for position bias.
• Elasticsearch and OpenSearch LTR plugins let you deploy trained models directly in the search engine, scoring documents during query execution without external latency.
• Effective features combine document-level signals (BM25, freshness, popularity) with behavioral signals (click rates, dwell time, user personalization).
• Data quality—especially avoiding biased training sets and ensuring sufficient click coverage—is more important than model complexity. Audit your logs before building.
• Start simple, measure impact, then iterate. Retraining models regularly keeps rankings aligned with evolving user preferences.