Improving RAG retrieval: AI & ML lessons from production

Improving RAG retrieval: AI & ML lessons from production

When your retrieval-augmented generation system starts returning wrong answers, the problem almost never lies in the language model itself—it lives in retrieval. We've seen this pattern repeatedly across e-commerce search, customer support, and SaaS documentation systems: a well-tuned retriever beats a larger model almost every time. This article walks through four battle-tested techniques we've used to transform retrieval quality in production environments.

Why retrieval failures compound

RAG systems fail silently. Your LLM confidently synthesizes an answer from poor source documents, and users only discover the problem after acting on bad information. A customer support agent hands a wrong policy interpretation to a customer. A product recommendation engine suggests inventory that doesn't match the query. These aren't hallucinations in the traditional sense—they're retrieval failures that propagate downstream.

The root cause usually sits in one of three places: keyword-only search misses semantic intent, vector search alone drifts into irrelevant regions of embedding space, or top results are noisy and require ranking. We'll address each.

Hybrid keyword + vector search

Single-modality retrieval is a liability. Keyword search excels at exact matches and rare terms; vector search captures intent and paraphrase. Combining both typically improves recall by 15–30% in our systems.

The strategy: run both searches in parallel, merge results, and deduplicate. Keyword search catches factual queries like "what is SKU 12345?" while vector search retrieves "what products go well with winter jackets?" Your merged result set gives the reranker more options to work with.

from elasticsearch import Elasticsearch
import numpy as np
from sentence_transformers import SentenceTransformer

class HybridRetriever:
    def __init__(self, es_client, model_name="all-MiniLM-L6-v2"):
        self.es = es_client
        self.model = SentenceTransformer(model_name)
    
    def retrieve(self, query, top_k=10):
        # Keyword search via Elasticsearch
        keyword_results = self.es.search(
            index="documents",
            body={
                "query": {
                    "multi_match": {
                        "query": query,
                        "fields": ["title^2", "content"]
                    }
                },
                "size": top_k
            }
        )
        
        # Vector search
        query_embedding = self.model.encode(query)
        vector_results = self.es.search(
            index="documents",
            body={
                "query": {
                    "script_score": {
                        "query": {"match_all": {}},
                        "script": {
                            "source": "cosineSimilarity(params.query_vector, 'embedding') + 1.0",
                            "params": {"query_vector": query_embedding.tolist()}
                        }
                    }
                },
                "size": top_k
            }
        )
        
        # Merge and deduplicate
        seen = {}
        merged = []
        for source in [keyword_results, vector_results]:
            for hit in source["hits"]["hits"]:
                doc_id = hit["_id"]
                if doc_id not in seen:
                    seen[doc_id] = hit
                    merged.append(hit)
        
        return merged[:top_k]

In production, we've found that weighting keyword results slightly higher (1.2x multiplier on their scores) helps when dealing with technical documentation where exact terminology matters. Adjust the weighting based on your domain—e-commerce often favors vector search more heavily, while legal or compliance systems lean keyword-heavy.

Rerank top results with a cross-encoder

Retrievers are retrieval-optimized; rerankers are ranking-optimized. A cross-encoder like BAAI/bge-reranker-large takes a query and document pair directly and produces a relevance score. Unlike bi-encoders (which independently embed query and document), cross-encoders see both inputs together and typically improve top-1 accuracy by 20–40%.

The flow: retrieve top 20–30 candidates, rerank with a cross-encoder, return top 3–5 for the LLM. This adds ~100–200ms latency but transforms answer quality.

from sentence_transformers import CrossEncoder

class RerankedRetriever:
    def __init__(self, hybrid_retriever, reranker_model="BAAI/bge-reranker-large"):
        self.retriever = hybrid_retriever
        self.reranker = CrossEncoder(reranker_model)
    
    def retrieve_and_rerank(self, query, candidate_k=20, final_k=5):
        # Get candidate documents
        candidates = self.retriever.retrieve(query, top_k=candidate_k)
        
        # Prepare pairs for reranking
        passages = [hit["_source"]["content"] for hit in candidates]
        pairs = [[query, passage] for passage in passages]
        
        # Rerank
        scores = self.reranker.predict(pairs)
        
        # Sort by reranker scores
        ranked = sorted(
            zip(candidates, scores),
            key=lambda x: x[1],
            reverse=True
        )
        
        return [doc for doc, score in ranked[:final_k]]

We've deployed cross-encoder reranking across customer support systems and saw a measurable drop in "irrelevant answer" user complaints. The cost—a few hundred milliseconds—is worth it when correctness matters.

Rewrite queries before searching

User queries are messy. They're often vague, use colloquial language, or bundle multiple sub-questions. Query rewriting uses an LLM to rephrase the original into a clearer, search-optimized form. This is computationally cheap (a small model works fine) and surprisingly effective.

For example, "do you have that blue thing from the sale last week?" becomes "blue product discount recent" or better yet, multiple queries: "blue products", "sale items last 7 days", and the original. You then retrieve for all and merge results.

from openai import OpenAI

class QueryRewriter:
    def __init__(self, client=None):
        self.client = client or OpenAI()
    
    def rewrite_queries(self, user_query):
        prompt = f"""Rewrite this search query into 2-3 clearer, more specific queries that would retrieve relevant documents.
Return only the rewrites, one per line. Do not include explanations.

Original query: {user_query}"""
        
        response = self.client.chat.completions.create(
            model="gpt-3.5-turbo",
            messages=[{"role": "user", "content": prompt}],
            temperature=0.3,
            max_tokens=150
        )
        
        rewrites = response.choices[0].message.content.strip().split("\n")
        return [user_query] + [r.strip() for r in rewrites if r.strip()]

class EnhancedRetriever:
    def __init__(self, hybrid_retriever, query_rewriter):
        self.retriever = hybrid_retriever
        self.rewriter = query_rewriter
    
    def retrieve(self, query, top_k=10):
        queries = self.rewriter.rewrite_queries(query)
        
        all_results = {}
        for q in queries:
            results = self.retriever.retrieve(q, top_k=top_k)
            for hit in results:
                doc_id = hit["_id"]
                if doc_id not in all_results:
                    all_results[doc_id] = hit
        
        return list(all_results.values())[:top_k]

Query rewriting shines in support and marketplace contexts where users phrase the same request dozens of ways. We've used it to improve retrieval for rare product variants (users describe them, the rewriter standardizes into sku or model number) and multi-language systems (translate, normalize, then retrieve).

Filter by metadata like product or date

Not all retrieved documents are equal. Metadata filters—category, date range, product line, price tier—let you narrow the retrieval space before ranking. This is especially powerful when combined with hybrid search, because a keyword match on an old, deprecated product can be filtered out early.

Implement filtering at the retriever layer, before ranking:

class FilteredRetriever:
    def __init__(self, hybrid_retriever):
        self.retriever = hybrid_retriever
    
    def retrieve_with_filters(self, query, filters=None, top_k=10):
        # filters: {"product_category": "electronics", "created_after": "2024-01-01"}
        
        # For Elasticsearch, add filter clause
        body = {
            "query": {
                "bool": {
                    "must": [
                        {
                            "multi_match": {
                                "query": query,
                                "fields": ["title^2", "content"]
                            }
                        }
                    ]
                }
            },
            "size": top_k
        }
        
        # Add metadata filters
        if filters:
            filter_clauses = []
            if "product_category" in filters:
                filter_clauses.append(
                    {"term": {"category": filters["product_category"]}}
                )
            if "created_after" in filters:
                filter_clauses.append(
                    {"range": {"created_at": {"gte": filters["created_after"]}}}
                )
            if "price_max" in filters:
                filter_clauses.append(
                    {"range": {"price": {"lte": filters["price_max"]}}}
                )
            
            if filter_clauses:
                body["query"]["bool"]["filter"] = filter_clauses
        
        results = self.retriever.es.search(index="documents", body=body)
        return results["hits"]["hits"]

Filtering is contextual. A user asking "winter coats under $200" should only see current inventory, not archived listings. A FAQ system querying "how do I reset my password?" should prioritize recent documentation over six-year-old posts. Always ask: what metadata would a human use to narrow this search?

Common mistakes

• Skipping the retriever problem entirely. Many teams jump to prompt engineering or model swaps when retrieval is the real bottleneck. Measure retrieval quality first (precision, recall on a held-out test set) before tuning the LLM.
• Reranking too few candidates. If you rerank the top 5, you're likely reranking a narrow slice. Retrieve 20–30, then rerank down to 5. The extra retrieval cost is negligible; the quality gain is huge.
• Using the same embedding model for both retrieval and reranking. Cross-encoders are a different architecture; they see query and document together and typically outperform bi-encoders on ranking. Don't replace them with more bi-encoders.
• Ignoring latency. Query rewriting and reranking add time. In real-time systems (chat, search), measure end-to-end latency. Sometimes sacrificing 5% retrieval quality for 2x speed is the right call.
• Not testing on real user queries. Your test queries might skew toward what your system already handles well. Periodically sample actual user queries and measure retrieval quality on those. This is where real gaps appear.

Key takeaways

• Hybrid retrieval (keyword + vector) consistently outperforms either modality alone; run both in parallel and merge results.
• Cross-encoder reranking on top-20 candidates lifts top-1 accuracy significantly and is worth the 100–200ms latency in most production scenarios.
• Query rewriting with a small LLM normalizes messy user input and retrieves from multiple reformulations, especially effective for vague or colloquial queries.
• Metadata filters (category, date, price) reduce noise before ranking and let you enforce business logic (e.g., only current inventory) at retrieval time.
• Measure retrieval quality separately from generation quality; most RAG failures stem from retrieval, not the language model.