Building a retrieval-augmented generation (RAG) system is where search and LLMs meet in production. Elasticsearch for RAG: a practical guide for developers shows you how to move beyond simple keyword search and build systems that actually work when your model needs grounded, verifiable answers.
Why Elasticsearch matters for RAG
RAG systems solve a real problem: LLMs hallucinate. By retrieving relevant documents before prompting, you ground responses in actual data. But retrieval quality directly impacts output quality. A search system that misses relevant documents or returns noise will feed garbage into your prompt, and no amount of prompt engineering fixes that.
Elasticsearch handles the scale and flexibility you need. When you're chunking documents, storing embeddings, filtering by permissions, and ranking results, you need a system built for search at scale, not a database bolted on with vector extensions. We've seen RAG systems fail because they used generic databases; Elasticsearch handles the retrieval layer properly.
Chunking documents and storing embeddings
Before anything reaches your LLM, documents must become chunks, and chunks need embeddings. Your chunking strategy directly impacts RAG quality, so this isn't something to delegate to defaults.
Start by choosing a chunk size. Too small and you lose context; too large and you retrieve irrelevant material. We typically land on 512–1024 tokens per chunk for most use cases, with 20% overlap to maintain context across boundaries. For a SaaS platform ingesting support documentation, a 300-word chunk with 50-word overlap handles both search precision and LLM context windows reasonably well.
Here's a practical approach to chunking and indexing:
{
"settings": {
"number_of_shards": 2,
"number_of_replicas": 1,
"index.knn": true,
"index.knn.space_type": "cosinesimilarity"
},
"mappings": {
"properties": {
"id": { "type": "keyword" },
"document_id": { "type": "keyword" },
"chunk_index": { "type": "integer" },
"text": { "type": "text", "analyzer": "standard" },
"embedding": {
"type": "dense_vector",
"dims": 1536,
"index": true,
"similarity": "cosine"
},
"metadata": {
"type": "object",
"properties": {
"source": { "type": "keyword" },
"document_type": { "type": "keyword" },
"created_at": { "type": "date" },
"owner_id": { "type": "keyword" }
}
},
"permissions": {
"type": "keyword"
}
}
}
}
When you ingest documents, generate embeddings using a model like OpenAI's text-embedding-3-small or an open-source alternative like nomic-embed-text. Store both the raw text and embedding. The text enables hybrid search (more on that next); the embedding powers vector similarity. In production, we batch embedding generation to stay within rate limits and reduce API costs.
Hybrid retrieval improves recall
Pure vector search feels elegant but misses real-world patterns. A user searching for "PostgreSQL backup" might use exact terminology; vector search alone might return results about "database snapshots" or "recovery procedures," which are related but not what they asked for. Pure keyword search returns exact matches but struggles with synonyms and semantic understanding.
Hybrid retrieval combines both. Query the index twice—once with keyword search, once with vector similarity—then merge and rank the results. You get both precision and recall.
A practical hybrid query looks like this:
{
"query": {
"bool": {
"should": [
{
"multi_match": {
"query": "PostgreSQL backup strategy",
"fields": ["text^2", "metadata.source"],
"fuzziness": "AUTO",
"operator": "OR"
}
},
{
"knn": {
"embedding": {
"vector": [/* embedding vector from query */],
"k": 10
}
}
}
],
"filter": [
{ "terms": { "permissions": ["user-team-1", "public"] } }
]
}
},
"size": 20,
"track_scores": true
}
Run both branches. The keyword search catches exact terminology; the KNN search catches semantic meaning. Elasticsearch merges results by relevance score. You can adjust the weight of each branch by modifying boost values on the multi_match clause if keyword results consistently rank too high or too low in your domain.
In a trading platform, we found hybrid retrieval cut false negatives by 60%. Pure vector search missed regulatory documents that used specific legal terminology; pure keyword search missed references to similar scenarios. Together, they worked.
Filter by permissions
Retrieval in multi-tenant systems must enforce permissions. A customer support agent should never see another customer's data, even if it's semantically relevant. Build permission filtering into your query, not as a post-processing step.
Use the filter clause in your bool query. In the example above, we filter on permissions before scoring. This ensures results can't leak across tenant boundaries regardless of retrieval method.
Permission schemas vary. For SaaS, you might use team IDs:
{
"filter": [
{ "terms": { "permissions": ["team-123", "public", "admin"] } }
]
}
For role-based access, combine multiple permission fields:
{
"filter": [
{
"bool": {
"should": [
{ "term": { "visibility": "public" } },
{ "term": { "owner_id": "user-456" } },
{ "terms": { "allowed_roles": ["viewer", "editor"] } }
],
"minimum_should_match": 1
}
}
]
}
At query time, always populate the permissions array from your auth context. Never trust client input. We've seen production incidents where permissions filters were accidentally omitted during refactoring; enforce it at the query builder level so it's impossible to forget.
Return sources for citations
LLMs are powerful but unreliable. Users need to verify where answers come from. Always return source metadata alongside results so citations are straightforward.
Store source information in your metadata field:
{
"metadata": {
"source": "help-article-42",
"source_title": "Backup Strategies for PostgreSQL",
"source_url": "https://docs.example.com/backup",
"section": "Full Backups",
"last_updated": "2024-01-15"
}
}
When you return results to your LLM prompt, include this metadata. A simple approach:
{
"results": [
{
"text": "pg_basebackup is the recommended tool for full backups...",
"score": 0.87,
"source": {
"id": "help-article-42",
"title": "Backup Strategies for PostgreSQL",
"url": "https://docs.example.com/backup",
"updated": "2024-01-15"
}
}
]
}
In your prompt, format these as references the model can cite. After generation, your frontend displays citations linked to the original sources. This builds trust and lets users validate answers. In a CRM we worked on, adding citations reduced support tickets about answer accuracy by 40% because users could check sources themselves.
Common mistakes
Storing only embeddings, not text. Tempting to save space, but you lose the ability to do keyword search and you can't display retrieved content to users without fetching elsewhere. Store both.
Using default chunking without validation. Every domain chunks differently. Medical records need different boundaries than API documentation. Test chunk sizes on your actual data before going to production.
Forgetting permissions in the filter clause. Put it in the query logic, not post-processing. Otherwise it's easy to accidentally skip under load or during refactoring.
Ignoring embedding model drift. If you upgrade your embedding model, old embeddings become incompatible. Plan reindexing windows and test new models on a subset before full rollout.
Setting k (KNN results) too high. Retrieving 100 vectors to merge with keyword results bloats network traffic and scoring time. Start with k=10 and tune based on latency requirements.
Key takeaways
- Chunk documents strategically with overlap, then generate embeddings once at index time, not query time.
- Hybrid retrieval—keyword plus vector—catches both exact matches and semantic meaning, improving recall significantly.
- Filter by permissions in the query's filter clause to enforce data isolation; never rely on post-processing.
- Always return source metadata so citations are built-in, not retrofitted, and users can verify answers.
- RAG quality is determined by retrieval quality; spend time tuning your Elasticsearch queries before worrying about prompt engineering.
Comments
Loading comments…