Summarization with LLMs: AI & ML lessons from production

When you need to extract meaning from thousands of customer support tickets, lengthy contract documents, or weeks of meeting transcripts, you quickly realize that summarization with LLMs: AI & ML lessons from production is far more nuanced than hitting an "summarize" button. In this article, we walk through the patterns we've learned building summarization systems at scale, from chunking strategies to source tracking to validation workflows that catch real-world edge cases.

The real problem with summarization at scale

Text summarization sounds simple in theory: feed text to a model, get a shorter version back. In practice, we've found that reliable summarization requires intentional architecture decisions, not just API calls.

Your first decision is whether you're summarizing a single email (few hundred tokens) or a 200-page contract (tens of thousands of tokens). Context windows have grown, but they're not infinite, and costs scale with input tokens. We've seen teams burn through budgets by sending entire datasets to GPT-4 without thinking about chunking strategies or whether the summarization task even needs a frontier model.

The second realization is that summarization quality depends heavily on your setup: prompt clarity, model temperature, length constraints, and what you do with sources. A summary that loses critical liability clauses is worse than no summary at all.

Setting length and audience

Before writing a single prompt, define your constraints. How many words or tokens is acceptable? Who reads this summary—a lawyer, a customer, an internal dashboard? Different audiences need different styles and detail levels.

We've found it helpful to think in three tiers:

• Executive summary: 2–3 sentences, highest-level takeaways only.
• Technical summary: One paragraph, includes specifics like error codes, thresholds, or API changes.
• Full summary with sections: Multiple paragraphs grouped by topic, preserves nuance for compliance or archival.

Declare your target length in the prompt explicitly. Don't rely on model intuition. Here's a practical example:

def create_summary_prompt(document: str, audience: str, max_tokens: int) -> str:
    audience_guides = {
        "executive": "Assume the reader has 2 minutes. Omit technical jargon.",
        "technical": "Assume the reader is familiar with our platform. Include specific metrics, error codes, and thresholds.",
        "legal": "Assume the reader is a lawyer. Preserve exact liability language and conditions.",
    }
    
    return f"""
Summarize the following document for a {audience} audience.

Target length: {max_tokens} tokens (approximately {max_tokens // 4} words).

Style guide: {audience_guides.get(audience, "Be concise and clear.")}

Document:
{document}

Summary:
"""

We've seen teams skip this step and get summaries that are either too vague or inappropriately detailed. Be explicit about expectations in your prompt.

Summarizing long documents in chunks

When your source document exceeds your model's comfortable working window, chunking is not optional. We typically use a recursive strategy: summarize chunks first, then summarize the summaries (often called "map-reduce" summarization).

Here's the challenge: if you just split on word boundaries, you'll often cut summaries mid-sentence or create orphaned context. We chunk on semantic boundaries: paragraphs, sections, or sentence clusters with some overlap.

Example: a 50-page incident report gets split into sections (architecture, timeline, root cause, mitigation). Each section is summarized independently, then those summaries are combined into one master summary.

import anthropic

def chunk_document(text: str, chunk_size: int = 3000, overlap: int = 300) -> list[str]:
    """
    Split document into overlapping chunks at sentence boundaries.
    """
    sentences = text.replace("! ", "!|").replace("? ", "?|").replace(". ", ".|").split("|")
    chunks = []
    current_chunk = ""
    
    for sentence in sentences:
        if len(current_chunk) + len(sentence) > chunk_size and current_chunk:
            chunks.append(current_chunk)
            current_chunk = chunks[-1][-overlap:] if chunks else ""
        current_chunk += sentence
    
    if current_chunk:
        chunks.append(current_chunk)
    
    return chunks

def summarize_chunk(client: anthropic.Anthropic, chunk: str, audience: str = "technical") -> str:
    """
    Summarize a single chunk using Claude.
    """
    message = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=500,
        messages=[
            {
                "role": "user",
                "content": f"""Summarize this section concisely for a {audience} audience.
Keep the summary under 300 words. Preserve any numbers, dates, or critical details.

Section:
{chunk}

Summary:"""
            }
        ]
    )
    return message.content[0].text

def summarize_document(client: anthropic.Anthropic, document: str, audience: str = "technical") -> str:
    """
    Summarize a long document using map-reduce pattern.
    """
    chunks = chunk_document(document, chunk_size=3000, overlap=300)
    
    chunk_summaries = []
    for chunk in chunks:
        summary = summarize_chunk(client, chunk, audience)
        chunk_summaries.append(summary)
    
    combined = "\n\n".join(chunk_summaries)
    
    final_message = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=800,
        messages=[
            {
                "role": "user",
                "content": f"""Consolidate these section summaries into one coherent summary.
Remove redundancy. Maintain chronological order where relevant.
Target: 500 words for a {audience} audience.

Section summaries:
{combined}

Final summary:"""
            }
        ]
    )
    return final_message.content[0].text

The key insight: don't treat the document as one big blob. Respect structure when it exists. If there's no natural structure, chunk by length but preserve context with overlaps.

Ask for key points with sources

A summary without sources is a liability risk in production. If you tell a user "the system went down on Tuesday," can you point back to the exact log line that says so? If not, your summary isn't trustworthy.

We ask the model to extract key points as a structured list, each paired with a quote or reference to the source document. This doesn't require custom training—a good prompt handles it.

def summarize_with_citations(client: anthropic.Anthropic, document: str) -> dict:
    """
    Return a summary with cited key points.
    """
    message = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=1000,
        messages=[
            {
                "role": "user",
                "content": f"""Summarize this document and list key points with exact citations.

For each key point, include:
1. The point (one sentence)
2. A direct quote from the source (in quotes)
3. The approximate location (e.g., "paragraph 3", "section titled Financials")

Format:
Key point 1: [statement]
Quote: "[exact text from document]"
Location: [where in the document]

Document:
{document}

Key points with citations:"""
            }
        ]
    )
    
    raw_response = message.content[0].text
    
    return {
        "summary": raw_response,
        "model": "claude-3-5-sonnet-20241022",
        "input_tokens": message.usage.input_tokens,
        "output_tokens": message.usage.output_tokens,
    }

In production, we store both the summary and the citation metadata in our database. When a user questions a summary, we can surface the source text immediately. This builds trust and reduces support burden.

Check for missing critical facts

Even the best summarization can miss critical details—especially edge cases, caveats, or buried context that changes the meaning of the summary.

We use a multi-pass validation approach:

1. Automated checklist: Did the summary mention dates, amounts, or names you expect to see?
2. Contradiction detection: Does the summary align with the original document? Have facts been inverted or misrepresented?
3. Sampling review: Humans spot-check a percentage of summaries, especially for high-stakes domains (legal, healthcare, finance).

Here's a pattern we use to detect potential hallucinations or omissions:

def validate_summary(client: anthropic.Anthropic, original: str, summary: str, critical_terms: list[str]) -> dict:
    """
    Check if summary contains critical information and is consistent with original.
    """
    validation_prompt = f"""Given the original document and its summary, answer these questions:

1. Does the summary mention all of: {', '.join(critical_terms)}?
2. Are there any contradictions between the summary and the original?
3. What important facts from the original are missing in the summary?

Original (excerpt):
{original[:2000]}

Summary:
{summary}

Analysis:"""
    
    message = client.messages.create(
        model="claude-3-5-sonnet-20241022",
        max_tokens=600,
        messages=[
            {
                "role": "user",
                "content": validation_prompt
            }
        ]
    )
    
    analysis = message.content[0].text
    
    missing_count = sum(1 for term in critical_terms if term.lower() not in summary.lower())
    
    return {
        "validation_analysis": analysis,
        "missing_critical_terms": missing_count,
        "requires_review": missing_count > 0 or "contradiction" in analysis.lower(),
    }

In our e-commerce and SaaS workflows, we flag any summary that fails this check for human review. It's not foolproof, but it catches most issues before they reach end users.

Common mistakes

• Ignoring token costs: Summarizing everything with GPT-4 sounds safe until your bill arrives. Use cheaper models for routine work; reserve expensive models for edge cases or audits.
• No length discipline: Leaving max_tokens unset invites rambling summaries. Always set explicit constraints.
• Chunking without overlap: Splitting a document blindly can orphan context. Overlapping chunks preserve continuity.
• Skipping validation: A confident-sounding summary that's wrong is worse than no summary. Build in spot-checking and contradiction detection.
• Losing source references: A summary without citations is hard to debug and risky in compliance-heavy domains. Always track where facts came from.
• One-shot summarization for long docs: If your document is 20+ pages, chunking and map-reduce will give you better results than a single prompt, even with a large context window.

Key takeaways

• Define your audience and target length explicitly in your prompt; don't rely on model defaults.
• Use map-reduce (chunk-summarize-combine) for documents longer than a few thousand tokens to maintain quality and control costs.
• Always request citations or source references; a summary without provenance is not production-ready.
• Implement validation layers—automated term checks, contradiction detection, and human spot-checks—especially for high-stakes domains.
• Choose your model tier intentionally; not every summarization task needs a frontier model, and chunking strategies let you use smaller models efficiently.
• Monitor summary quality over time; track user feedback and re-validate periodically as your source data changes.