RAG implementation tutorial: what developers should know

Building a chatbot or Q&A system that grounds answers in your own data used to require careful orchestration of retrieval and generation pipelines. Retrieval-augmented generation (RAG) has become the go-to pattern for letting an LLM answer questions from your company's documents, knowledge bases, and data stores without fine-tuning. This RAG implementation tutorial: what developers should know walks through the architecture, chunking strategies, source attribution, and evaluation methods that separate production systems from prototypes.

Understanding the RAG pipeline

RAG works in two distinct phases: retrieval first, then generation. The retriever finds the most relevant pieces of your data, and the generator (the LLM) crafts an answer using those pieces as context. This separation is powerful because it lets you swap out either component—use a different embedding model, vector database, or LLM—without rewriting the entire system.

The flow looks like this:

1. Document ingestion: Split your documents (PDFs, Markdown, database exports) into manageable chunks.
2. Embedding and storage: Convert each chunk into a dense vector and store it in a vector database alongside metadata (document name, page number, timestamp).
3. User question embedding: When a user asks a question, embed that question using the same embedding model.
4. Semantic retrieval: Find the most similar chunks by comparing vectors (usually via cosine similarity or dot product).
5. LLM generation: Pass the user's question plus the retrieved chunks to an LLM as context, and let it synthesize an answer.

In practice, this means your backend does a vector similarity search (often against Pinecone, Weaviate, or a local FAISS index) before calling the LLM. For a production e-commerce platform, imagine a customer asking "What's your return policy for items over $500?" The system retrieves the relevant policy chunks, then the LLM generates a natural, conversational answer grounded in those policies.

Chunk well: size, overlap, and boundaries

How you split documents into chunks affects both retrieval quality and LLM context quality. Too small, and you lose semantic coherence. Too large, and you waste tokens and dilute relevant information with noise.

Our experience across production systems suggests 300–800 tokens per chunk as a reliable sweet spot. For context, 300 tokens is roughly 200–250 words; 800 tokens is closer to 600 words. Use an accurate tokenizer for your embedding model—OpenAI's tiktoken for their models, or Hugging Face's tokenizers for open-source embeddings.

Always introduce overlap. A 50–100 token overlap between consecutive chunks prevents you from accidentally splitting important semantic units. If a sentence spans two chunks, the overlap ensures both chunks contain the full thought.

Split on semantic boundaries. Markdown headings are natural breakpoints. So are paragraph breaks in prose or section dividers in documentation. Avoid splitting mid-sentence or mid-concept. Here's a practical chunking strategy:

import tiktoken
from typing import List

def chunk_text(text: str, chunk_size: int = 500, overlap: int = 100) -> List[str]:
    """Split text into overlapping chunks by token count."""
    enc = tiktoken.encoding_for_model("gpt-3.5-turbo")
    tokens = enc.encode(text)
    chunks = []
    
    for i in range(0, len(tokens), chunk_size - overlap):
        chunk_tokens = tokens[i : i + chunk_size]
        chunks.append(enc.decode(chunk_tokens))
    
    return chunks

# For structured documents, split by headings first
def chunk_by_headings(markdown_text: str) -> List[str]:
    """Split markdown by h2/h3 headers, then chunk further if needed."""
    import re
    sections = re.split(r'^(## |### )', markdown_text, flags=re.MULTILINE)
    chunks = []
    
    for i in range(1, len(sections), 2):
        header = sections[i] + sections[i + 1] if i + 1 < len(sections) else sections[i]
        # If section is too long, recursively chunk it
        if len(header.split()) > 800:
            chunks.extend(chunk_text(header, chunk_size=500, overlap=100))
        else:
            chunks.append(header)
    
    return chunks

On a travel platform, you might chunk hotel descriptions by amenities sections (rooms, dining, policies), then apply token-based chunking to each section so no single chunk exceeds your LLM's context window.

Cite sources and make answers verifiable

An LLM can hallucinate plausible-sounding answers even when given good context. Users need to verify claims. Always attach metadata to each chunk so you can cite the source of each answer.

Store alongside your embedding vectors the chunk ID, document name, URL or file path, and optionally the page number or section header. When you retrieve chunks for the LLM, pass their IDs to the generation step. After the LLM generates an answer, your backend can reconstruct which chunks were used and return them to the user.

Here's a typical flow:

from dataclasses import dataclass

@dataclass
class Chunk:
    id: str
    text: str
    document: str
    url: str
    page: int = None

# After retrieval from vector DB
retrieved_chunks: List[Chunk] = vector_db.search(
    query_embedding, 
    top_k=5
)

# Build context for LLM
context = "\n".join([
    f"[{chunk.id}] {chunk.text}"
    for chunk in retrieved_chunks
])

# Call LLM
answer = llm.generate(
    prompt=f"Answer based on: {context}\n\nQuestion: {user_question}"
)

# Return answer with sources
return {
    "answer": answer,
    "sources": [
        {
            "id": chunk.id,
            "document": chunk.document,
            "url": chunk.url,
            "page": chunk.page
        }
        for chunk in retrieved_chunks
    ]
}

This is especially critical in regulated industries like financial services or healthcare, where traceability matters. A customer support system for an SaaS platform should cite the exact help article section that informed the answer.

Evaluate retrieval hit rate and answer quality

Building a RAG system is only half the battle. You need to measure whether it's working. Two key metrics:

Retrieval hit rate: For a set of test questions, did the retriever pull back at least one chunk that contains the answer? If you have 100 test questions and 85 of them have relevant chunks in the top 5 results, your hit rate is 85%. A hit rate below 80% usually signals chunking problems, inadequate embedding model quality, or insufficient data coverage.

Answer quality: Once chunks are retrieved, does the LLM generate a correct, complete answer? This requires human evaluation or LLM-as-judge techniques. You can use a rubric: Does the answer directly address the question? Is it grounded in the retrieved chunks? Are any claims unsupported by the sources?

Build a small evaluation set from real user questions and expected answers. Run your RAG system on these questions monthly or after major changes:

from enum import Enum

class AnswerQuality(Enum):
    CORRECT = 1
    PARTIAL = 0.5
    INCORRECT = 0

def evaluate_rag(questions: List[str], expected_answers: List[str]) -> dict:
    """Evaluate retrieval and answer quality on a test set."""
    hit_count = 0
    quality_scores = []
    
    for question, expected in zip(questions, expected_answers):
        # Retrieve
        chunks = retrieve(question, top_k=5)
        if any(expected.lower() in chunk.text.lower() for chunk in chunks):
            hit_count += 1
        
        # Generate
        answer = generate(question, chunks)
        
        # Score (manual review or LLM-based)
        quality = evaluate_answer(answer, expected)
        quality_scores.append(quality)
    
    hit_rate = hit_count / len(questions)
    mean_quality = sum(quality_scores) / len(quality_scores)
    
    return {
        "retrieval_hit_rate": hit_rate,
        "mean_answer_quality": mean_quality,
        "total_questions": len(questions)
    }

In a ride-hailing platform, for example, you'd test questions like "How do I contest a fare?" against a curated set of expected answers. A retrieval hit rate of 90% and answer quality of 0.85 (on a 0–1 scale) indicates a healthy system; below 75% hit rate suggests you need better chunks or more comprehensive documentation coverage.

Common mistakes

• Chunks too large: Passing 2000+ tokens per chunk to the LLM wastes context and dilutes signal. Keep it tight.
• No overlap: Splitting without overlap risks breaking semantic units across chunk boundaries, making retrieval miss relevant context.
• Ignoring embedding quality: A weak embedding model (fine-tuned on unrelated data) will fail at semantic similarity. Match embedding model to your domain when possible; evaluate alternatives before committing to production.
• Not testing retrieval independently: Developers often blame the LLM when the real problem is poor retrieval. Always measure hit rate separately.
• Skipping source citation: Users rightly distrust answers without sources. Build citation into your data pipeline from day one; retrofitting is painful.
• Single-pass retrieval: For complex questions, one retrieval pass may not find all relevant chunks. Consider iterative retrieval, query expansion, or hybrid approaches (keyword + semantic search).
• Stale data: RAG systems degrade as underlying documents change. Plan for regular re-indexing and versioning of your vector database.

Key takeaways

• RAG decouples retrieval from generation, letting you optimize each independently. Retrieve first, then generate.
• Chunk documents into 300–800 token pieces with semantic boundaries (headings) and 50–100 token overlap.
• Always store and return chunk metadata (document name, URL, page) so answers are verifiable and trustworthy.
• Measure retrieval hit rate and answer quality on a curated test set; aim for 85%+ hit rate and mean quality above 0.8.
• Embedding model quality and chunking strategy matter more than LLM size for RAG performance. Invest in evaluation early.
• Plan for re-indexing and data freshness; RAG systems degrade as underlying documents age without updates.