Chunking strategies for RAG: what developers should know

We've all built a RAG system that seemed perfect in isolation—until users started querying it. The bottleneck wasn't your retriever or LLM; it was how you split your documents into chunks. Chunking strategies for RAG: what developers should know is actually the difference between a system that finds relevant context and one that returns noise. Getting this right early saves weeks of tuning later.

Why chunking matters more than you think

Chunking is where the "R" in RAG lives. Your embedding model, vector database, and LLM are only as good as the chunks you feed them. A poorly chunked document creates two problems: either your retriever pulls irrelevant fragments (false positives), or it misses the right answer because it was split across chunk boundaries.

In production e-commerce systems, we've seen chunk size differences shift retrieval accuracy by 20–30%. A chunk that's too small loses narrative context. A chunk that's too large drowns the signal in noise. The sweet spot depends entirely on your domain, query patterns, and content structure.

The goal is simple: each chunk should be a self-contained unit of meaning that answers one logical question or covers one coherent topic. Metadata and overlap are your allies here.

Split on structure: headings and paragraphs

The best chunking strategy respects the document's natural hierarchy. Most content—technical docs, product manuals, blog posts, even support articles—is already organized by headings and logical sections. Splitting against the grain creates misalignment and context loss.

Start by identifying structural boundaries: H1, H2, H3 headings, section breaks, even lists. Then chunk at paragraph or sub-section level within those boundaries. This preserves semantic structure and makes retrieval more predictable.

Here's a practical approach we use in our SaaS platforms:

from typing import List, Dict

def chunk_by_structure(text: str, heading_level: int = 2) -> List[Dict[str, str]]:
    """
    Split text by markdown headings while preserving hierarchy.
    Returns list of dicts with 'content', 'heading', and 'level'.
    """
    chunks = []
    current_section = None
    current_content = []
    current_level = None
    
    lines = text.split('\n')
    
    for line in lines:
        # Check for heading markers
        heading_count = 0
        for char in line:
            if char == '#':
                heading_count += 1
            else:
                break
        
        if heading_count > 0 and heading_count <= heading_level:
            # Save previous chunk if it exists
            if current_content and current_section:
                chunks.append({
                    'content': '\n'.join(current_content).strip(),
                    'heading': current_section,
                    'level': current_level
                })
            
            # Start new section
            current_section = line.replace('#', '').strip()
            current_level = heading_count
            current_content = []
        else:
            # Accumulate content
            if line.strip():
                current_content.append(line)
    
    # Don't forget the last chunk
    if current_content and current_section:
        chunks.append({
            'content': '\n'.join(current_content).strip(),
            'heading': current_section,
            'level': current_level
        })
    
    return chunks

For HTML or non-markdown content, parse the DOM tree and split on tag boundaries (div sections, article containers). The principle is identical: respect the author's structure. A heading followed by three paragraphs should stay together as a chunk, not be split arbitrarily at word count.

Real example: a product documentation system chunking by H2 headings with a max of 500 words per chunk. Each chunk inherits its parent H1 and H2 as metadata. Retrieval accuracy improved 28% just from this structural respect compared to naive 300-token splitting.

Overlap chunks to keep context

Chunking creates a problem: boundary loss. Information relevant to answering a query might live in the crack between two chunks. A query about "authentication workflow" might need context from both the "Authentication" section's end and the "Security" section's beginning.

Overlapping chunks solve this. Take the end of one chunk and prepend it to the start of the next. Or, keep the heading from the previous section as context at the top of the next chunk. This trades storage and retrieval time for recall.

We typically use 20–30% overlap (measured by word count or tokens). Here's how to implement it:

def add_chunk_overlap(chunks: List[str], overlap_percent: float = 0.25) -> List[str]:
    """
    Add overlapping context from previous chunk to each new chunk.
    overlap_percent: fraction of previous chunk to prepend.
    """
    if not chunks:
        return []
    
    overlapped = [chunks[0]]  # First chunk unchanged
    
    for i in range(1, len(chunks)):
        prev_chunk = chunks[i - 1]
        curr_chunk = chunks[i]
        
        # Calculate overlap size
        overlap_words = int(len(prev_chunk.split()) * overlap_percent)
        overlap_text = ' '.join(prev_chunk.split()[-overlap_words:])
        
        # Prepend overlap to current chunk
        merged = overlap_text + ' ' + curr_chunk
        overlapped.append(merged)
    
    return overlapped

In practice: a 1000-word guide chunked into four 300-word pieces. Without overlap, some queries missed the section transition entirely. With 25% overlap (75 words), those edge-case queries started hitting. Storage increased by ~12%, but recall lifted 8–12% depending on query type.

The tradeoff is real though. Overlapping increases vector database size and retrieval latency slightly. Only use it if you've measured a recall gap at chunk boundaries.

Store titles and metadata with each chunk

Chunks are not isolated. They come from somewhere. That lineage—source document, section heading, timestamp, URL—is crucial both for retrieval scoring and for providing context to the LLM (and users).

Store at minimum:

• Source document ID: Which file or page did this come from?
• Section hierarchy: Parent headings (H1, H2, even H3).
• Chunk index: Position within the document (for ordering or deduplication).
• Creation/update timestamp: When was this content added or modified?

Better systems also include domain-specific metadata: author, product version, customer tier, or content category. This enables filtering at retrieval time ("only show me docs for v2.0 of our API").

When storing chunks in a vector database, structure your metadata payload cleanly:

def create_chunk_record(
    content: str,
    source_doc_id: str,
    section_title: str,
    parent_heading: str,
    chunk_index: int,
    metadata: dict = None
) -> dict:
    """
    Create a chunk record ready for vector DB storage.
    """
    record = {
        'content': content,
        'metadata': {
            'source_doc_id': source_doc_id,
            'section_title': section_title,
            'parent_heading': parent_heading,
            'chunk_index': chunk_index,
            'url': f'/docs/{source_doc_id}#{section_title.replace(" ", "-")}',
            'created_at': datetime.utcnow().isoformat(),
        }
    }
    
    # Merge any additional metadata
    if metadata:
        record['metadata'].update(metadata)
    
    return record

When your LLM retrieves these chunks, it sees not just the fragment but also knows it came from "API Reference > Authentication > OAuth2 Flow". That context improves answer quality. And when you show results to users, you can cite the exact section, building trust.

Test chunk sizes against real queries

Theory only goes so far. The right chunk size for a technical API reference differs from a customer support FAQ. The only reliable way to optimize is testing against actual query patterns.

Set up a simple evaluation loop:

1. Collect 20–50 representative queries from users (or synthesize them).
2. For each query, manually identify which documents/sections should be retrieved (your ground truth).
3. Run your chunking + retrieval pipeline with different chunk sizes (e.g., 256 tokens, 512, 1024).
4. Measure recall (did we retrieve the right chunk?) and precision (how much noise did we pull?).
5. Pick the size that balances both for your use case.

Example: testing with a SaaS product documentation system. We started with 512-token chunks (industry default). Testing revealed:

For this domain, 512 won. Too small, and context suffered. Too large, and we retrieved irrelevant material. Importantly, this will differ for your content and queries—testing is non-negotiable.

Automate this. Store results. Re-test when you add significant new content or your query patterns shift. Chunking isn't a one-time decision; it's a tuning parameter.

Common mistakes

Mistake 1: Ignoring document structure. Chunking on fixed token counts without regard for heading boundaries creates misaligned chunks that lose context. Respect the content's natural organization.

Mistake 2: Never testing on real queries. A chunk size that performs well on synthetic data might fail on production queries. Always validate against actual usage patterns.

Mistake 3: Storing chunks without metadata. When your LLM needs to cite a source or when filtering is needed, missing metadata forces re-indexing or manual lookup. Embed it from day one.

Mistake 4: Overlap without measurement. Overlapping sounds good in theory but adds cost. Only use it if evaluation shows a recall gap at boundaries. Otherwise, it's waste.

Mistake 5: Static chunking forever. Content changes, query patterns evolve. Re-evaluate chunking strategy quarterly or after major content updates. What worked at launch may not work at scale.

Key takeaways

• Chunking quality directly controls retrieval quality; it's the foundation of RAG performance.
• Respect document structure by splitting on headings and sections, not arbitrary token counts.
• Add 20–30% chunk overlap only if testing shows a recall gap at boundaries.
• Store source metadata (document ID, heading hierarchy, timestamp, URL) with every chunk for context and filtering.
• Test chunk sizes against real user queries; the optimal size is domain and query-pattern specific.
• Re-evaluate chunking strategy periodically as content and usage evolve.