Tokens and context windows: patterns, pitfalls and fixes

Tokens and context windows: patterns, pitfalls and fixes

When you work with large language models in production, token limits feel like an invisible tax on every request. You're not sending words to the API—you're sending tokens, and those tokens have both financial and quality costs. Understanding how tokens and context windows work isn't just about staying within budget; it's about building systems that actually perform well at scale.

Tokens are word pieces, not words

The first misconception we see on real teams is treating tokens as synonymous with words. They're not. A token is a subunit of text, and the mapping is messy enough that it trips up developers who don't look closely.

Most modern LLMs use tokenizers that split text into character sequences. Common tokens include common words ("hello"), punctuation (" ."), partial words ("ing"), and even individual characters. The token count for a given string depends entirely on the tokenizer the model uses. OpenAI's GPT models use a different tokenizer than Claude, which uses a different one from Llama. The same sentence might be 12 tokens in one model and 15 in another.

In practice, a rough rule of thumb is that 1 token ≈ 4 characters or 0.75 words. So if you're planning for a 4,000-token context window and you have 1,000 words of context, you're safe. But that rule breaks down fast with code, special characters, and non-English text.

import tiktoken

# OpenAI tokenizer for GPT-3.5 / GPT-4
encoding = tiktoken.encoding_for_model("gpt-4")

text = "Understanding token limits is critical for production AI systems."
tokens = encoding.encode(text)
print(f"Text: {text}")
print(f"Token count: {len(tokens)}")
# Output: Token count: 12

# Same text, very different token count for code
code_snippet = """
def calculate_total(items):
    return sum([item['price'] * item['quantity'] for item in items])
"""
code_tokens = encoding.encode(code_snippet)
print(f"Code snippet tokens: {len(code_tokens)}")
# Output: Code snippet tokens: 41

Always tokenize your actual input before sending it to an API. Don't guess. Use the tokenizer library provided by the model vendor. It takes 30 seconds and prevents embarrassing overages or truncation bugs.

Context includes prompt and answer

The context window is the total number of tokens you can send and receive in a single request-response cycle. It includes your prompt, any system instructions, conversation history, retrieved documents, and the entire response the model generates.

Many teams make the mistake of thinking the context window is just "how much I can ask." It's not. If your model supports a 128K token window and you send 100K tokens of context, the model only has 28K tokens left to respond. A long answer gets truncated or incomplete.

Consider a customer support scenario where you're building a retrieval-augmented generation (RAG) system:

import openai

def answer_support_query(query, customer_history, product_docs):
    # System prompt
    system = "You are a helpful support agent. Answer based on provided context."
    
    # User query
    user_prompt = f"Customer question: {query}"
    
    # Retrieved context (this can get large fast)
    context = f"""
    Customer history:
    {customer_history}
    
    Product documentation:
    {product_docs}
    """
    
    # Full prompt tokens
    full_prompt = system + user_prompt + context
    tokens_used = estimate_tokens(full_prompt)  # Let's say 60K tokens
    
    # Model has context_limit - tokens_used left for the response
    # With 128K limit and 60K used, only 68K remain
    # But if response needs 50K, you only have 18K of safety margin
    
    response = openai.ChatCompletion.create(
        model="gpt-4-turbo",
        messages=[
            {"role": "system", "content": system},
            {"role": "user", "content": user_prompt + context}
        ],
        max_tokens=50000  # Explicit limit is good practice
    )
    
    return response

The pattern here matters: you're not just tracking input tokens, you're budgeting for output. In production, we set max_tokens explicitly to ensure the model stops before context exhaustion.

Long context costs more and can reduce focus

There are two costs to long context: literal cost and quality degradation.

Literal cost: Most LLM APIs charge per token. Longer prompts mean higher bills. At scale—processing 10,000 support tickets a day with 50K tokens of context each—the difference between careful context selection and dumping everything matters. We've seen teams cut their LLM spend by 30–40% just by being disciplined about what they send.

Quality degradation: This is less obvious but more important. Research and real-world production testing show that LLMs don't perform as well with very long contexts. The model's attention mechanism can get diluted. Relevant information buried in a 100K token prompt gets less attention than when it's in a 10K prompt. This is called the "lost in the middle" problem—key facts in the middle of your context window get ignored.

For example, if you're building a legal document analysis system and you send 10 full contracts (80K tokens), the model might miss a critical clause that appears on page 4 of document 5. Send only the relevant sections (8K tokens), and the model catches it.

import openai

def analyze_documents_wrong(contracts):
    # Dump everything (80K tokens)
    prompt = "Analyze these contracts for risks:\n" + "\n\n".join(contracts)
    response = openai.ChatCompletion.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}]
    )
    return response

def analyze_documents_right(contracts, query):
    # Extract only relevant sections
    relevant_sections = []
    for contract in contracts:
        sections = extract_sections_by_relevance(contract, query)
        relevant_sections.extend(sections)
    
    # Now just 8K tokens
    prompt = f"Analyze for: {query}\n" + "\n".join(relevant_sections)
    response = openai.ChatCompletion.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        temperature=0.2  # Lower temperature for factual tasks
    )
    return response

The right approach is faster, cheaper, and more accurate.

Retrieve only what is relevant

This is where RAG systems earn their keep. The core idea: don't let the LLM see everything in your knowledge base. Query your vector database, database, or search index first. Get the top-k most relevant results. Send only those to the LLM.

In production, we typically:

1. Receive a user query
2. Embed the query (small, fast)
3. Search for the top 3–10 relevant documents or passages
4. Concatenate those results into context
5. Send prompt + retrieved context to the LLM

The retrieval step is cheap (milliseconds, pennies) compared to LLM inference. Spend time optimizing retrieval quality, and you save on LLM costs while improving accuracy.

import openai
from pinecone import Pinecone

def retrieve_and_answer(user_query, namespace="support-docs"):
    # Step 1: Embed the query
    query_embedding = openai.Embedding.create(
        input=user_query,
        model="text-embedding-3-small"
    )["data"][0]["embedding"]
    
    # Step 2: Search vector database
    pc = Pinecone(api_key="your-key")
    index = pc.Index("docs")
    results = index.query(
        vector=query_embedding,
        top_k=5,
        namespace=namespace,
        include_metadata=True
    )
    
    # Step 3: Build context from top results
    context = "\n".join([
        match["metadata"]["text"]
        for match in results["matches"]
        if match["score"] > 0.7  # Confidence threshold
    ])
    
    # Step 4: Send to LLM with limited, relevant context
    response = openai.ChatCompletion.create(
        model="gpt-4-turbo",
        messages=[
            {
                "role": "user",
                "content": f"Context:\n{context}\n\nQuestion: {user_query}"
            }
        ],
        max_tokens=1000
    )
    
    return response

The confidence threshold (score > 0.7) is tunable. Higher threshold = only very relevant results, but you might miss some. Lower threshold = more coverage, but noise increases. Test both and measure impact on your downstream accuracy metrics.

Common mistakes

• Not tokenizing before budgeting: Teams estimate token counts by word count and get surprised by overages. Always use the actual tokenizer.
• Forgetting the response budget: Sending 90K tokens of context and expecting a 30K token response in a 128K window leaves no room for error. Always reserve tokens for the response.
• Assuming more context = better answers: It doesn't. Dumping entire databases into prompts degrades quality and wastes money. Quality retrieval beats quantity every time.
• Not setting max_tokens: Without an explicit limit, models can ramble. Set a reasonable cap to avoid surprises and keep costs predictable.
• Retrieving without filters: If your retrieval returns 50 semi-relevant documents instead of 5 highly relevant ones, you've wasted tokens and confused the model. Use confidence scores and filters.

Key takeaways

• Tokens are subword units, not words. Use the model's tokenizer to count accurately; never guess.
• Context window is input plus output. Budget for both, and always set max_tokens explicitly.
• Long context is expensive (higher API costs) and often produces worse results (attention diffusion). Optimize for brevity and relevance.
• Retrieval-augmented generation (RAG) is the pattern: query your knowledge store, get top-k results, send only those to the LLM.
• In production, treat token counting as seriously as you treat error handling. Automated monitoring of token usage per request catches budget surprises early.