Temperature and sampling: a AI & ML guide for developers

Temperature and sampling: a AI & ML guide for developers

When you call a language model API, you're not just sending a prompt—you're also making choices about how the model generates its response. The two most critical levers you have are temperature and sampling parameters, which control how predictable or creative the output becomes. Understanding these settings is essential for building reliable AI features, whether you're generating summaries, brainstorming product names, or retrieving factual information.

Why temperature matters in production

Temperature is a parameter that scales the model's confidence scores before it selects the next token. Think of it as controlling how much the model "commits" to its top choices versus considering lower-probability alternatives.

At a temperature of 0, the model always picks the token with the highest probability. This makes output completely deterministic—run the same prompt twice and you'll get identical results. As temperature increases, the model becomes more willing to explore less likely continuations. At temperature 2, even tokens with very low probabilities might get selected.

In practice, this means:

• Temperature 0–0.3: Focused, predictable, best for extraction and factual tasks
• Temperature 0.7–0.9: Balanced, still coherent but with some variety
• Temperature 1.0–1.5: Creative, more exploration, higher risk of inconsistency
• Temperature 1.5+: Wild, often incoherent, rarely used in production

The key insight for developers: your choice of temperature should match your task, not your gut feeling. A customer service chatbot needs temperature 0.7, not 1.2, even if "more creative" sounds better.

Low temperature for factual and structured tasks

When you need reliable, repeatable outputs, keep temperature low. This covers most real-world production workloads: extracting structured data, classifying documents, summarizing factual content, or generating SQL queries.

Consider a customer support ticket classifier. You're extracting the category, priority, and suggested response from incoming emails. If the model is "creative" here, it might classify a billing issue as product feedback, or mark low-priority items as urgent. The cost of variation is too high.

import anthropic

client = anthropic.Anthropic()

ticket = """
Subject: Can't log into my account
Body: I've been trying for an hour. My password isn't working 
and I don't see a reset option. This is urgent—I have a 
presentation in an hour.
"""

message = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=200,
    temperature=0.2,  # Low temperature for consistency
    messages=[
        {
            "role": "user",
            "content": f"""Classify this support ticket. Return JSON only.
{ticket}

JSON format:
{{"category": "...", "priority": "low|medium|high", "suggested_response": "..."}}"""
        }
    ]
)

print(message.content[0].text)

With temperature 0.2, every time you run this prompt on the same ticket, you'll get the same classification. That's critical for consistent logging, routing, and metrics. You can measure your classifier's accuracy over time without temperature drift confounding your results.

The same principle applies to code generation. If you're using a model to generate database queries, API calls, or test scaffolding, keep temperature between 0 and 0.5. Creativity here often means broken syntax.

Higher temperature for brainstorming and exploration

Some tasks benefit from variety. Product naming, tagline generation, creative writing, or exploring multiple solution approaches all need the model to take risks and suggest less obvious alternatives.

The difference from extraction tasks is that you don't need every output to be identical, and in fact you want divergence. When brainstorming, you're generating multiple candidates and humans filter later.

import anthropic

client = anthropic.Anthropic()

message = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=500,
    temperature=1.2,  # Higher for variety
    messages=[
        {
            "role": "user",
            "content": """Generate 5 creative product names for an AI-powered 
note-taking app focused on developers. Names should be memorable, 
tech-forward, but not cliché. Return one name per line."""
        }
    ]
)

print(message.content[0].text)

Run this three times with temperature 1.2 and you'll get meaningfully different suggestions each time. Run it with temperature 0.2 and you'll get nearly identical output. For brainstorming, identical is boring.

The trade-off is that higher temperature sometimes produces lower-quality output. A name might be awkward or already trademarked. That's why brainstorming with models works best when you generate multiple candidates and filter or refine them afterward, rather than expecting a single perfect answer.

Understanding top_p (nucleus sampling)

While temperature controls the shape of the probability distribution, top_p (also called nucleus sampling) limits which tokens are even considered. It works by sorting tokens by probability and only sampling from the top candidates that collectively account for top_p percent of the probability mass.

If top_p is 0.9, the model considers tokens that account for 90% of the probability. Everything else is filtered out. This is orthogonal to temperature:

• Temperature adjusts how much the model leans toward its top choices
• top_p removes low-probability tokens entirely

In practice, top_p is gentler than temperature. A combination of temperature 0.8 and top_p 0.95 produces coherent, varied output. Temperature 2.0 and top_p 1.0 produces nonsense.

import anthropic

client = anthropic.Anthropic()

# Variant 1: Temperature only
message1 = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=100,
    temperature=0.9,
    top_p=1.0,  # Consider all tokens
    messages=[{"role": "user", "content": "Write a haiku about debugging."}]
)

# Variant 2: Temperature + top_p
message2 = client.messages.create(
    model="claude-3-5-sonnet-20241022",
    max_tokens=100,
    temperature=0.9,
    top_p=0.9,  # Only top 90% of probability
    messages=[{"role": "user", "content": "Write a haiku about debugging."}]
)

print("Temperature only:", message1.content[0].text)
print("Temperature + top_p:", message2.content[0].text)

Most developers don't need to tune top_p carefully. Set it to 0.95 or 1.0 and focus on temperature. The exception is when you want to be especially conservative: medical diagnosis, legal document review, or compliance tasks might use top_p 0.8 to exclude unlikely candidates entirely.

Keep settings fixed when evaluating

This is the most commonly overlooked mistake in AI development. When you're measuring model quality, accuracy, latency, or cost, your sampling parameters must stay constant across runs. Otherwise you can't tell if improvements came from your prompt tweaks or just from randomness.

Imagine you're evaluating a new version of your extraction prompt. You run it 10 times with the old prompt, then 10 times with the new one. But you accidentally use temperature 0.3 for the old version and temperature 0.7 for the new one. The new version looks better, but that's just because higher temperature produces more variable output, and you got lucky with the random seed.

Set up a test harness that logs temperature, top_p, seed, and any other sampling parameters alongside your results:

import anthropic
import json
from datetime import datetime

client = anthropic.Anthropic()

def evaluate_prompt(prompt_text, version, temperature, top_p, num_runs=5):
    results = []
    
    for run in range(num_runs):
        message = client.messages.create(
            model="claude-3-5-sonnet-20241022",
            max_tokens=200,
            temperature=temperature,
            top_p=top_p,
            messages=[{"role": "user", "content": prompt_text}]
        )
        
        result = {
            "timestamp": datetime.utcnow().isoformat(),
            "version": version,
            "run": run + 1,
            "temperature": temperature,
            "top_p": top_p,
            "output": message.content[0].text,
            "usage": {
                "input_tokens": message.usage.input_tokens,
                "output_tokens": message.usage.output_tokens
            }
        }
        results.append(result)
    
    return results

# Always use the same settings across evaluation runs
old_results = evaluate_prompt("Extract the priority level.", "v1", 
                               temperature=0.3, top_p=1.0)
new_results = evaluate_prompt("Extract the priority level from this ticket.", "v2", 
                               temperature=0.3, top_p=1.0)

# Save for analysis
with open("eval_results.json", "w") as f:
    json.dump({"old": old_results, "new": new_results}, f, indent=2)

When you document your model's behavior, always include the temperature and top_p settings. "This model has 94% accuracy" means nothing without "at temperature 0.2". At temperature 1.0, accuracy might drop to 87%.

Common mistakes

• Confusing temperature with quality: Higher temperature is not "better" or "smarter". It's just different. For most production tasks, lower is better.
• Varying sampling parameters during evaluation: Your benchmark results are only valid if temperature, top_p, and other settings remain constant. Otherwise you're measuring randomness, not model quality.
• Setting temperature too high for creative tasks: You don't need temperature 1.8 for brainstorming. Temperature 1.0–1.2 gives variety without incoherence. Anything above 1.5 usually produces noise.
• Not documenting your settings: Six months from now, you won't remember why a certain feature works well. Log temperature and top_p in your production systems and your experiment logs.
• Tuning temperature without context: Temperature should match your task, not your preference. A support chatbot should use lower temperature even if higher feels more "human-like" to you.

Key takeaways

• Temperature controls how predictable a model's output is. Low temperature (0–0.3) for factual tasks, higher (0.7–1.2) for creative ones.
• For extraction, classification, and code generation, prefer temperature 0.2–0.5 to ensure consistency and repeatability.
• For brainstorming and creative tasks, use temperature 1.0–1.2 to encourage variety, then filter or refine afterward.
• top_p limits which tokens the model considers, providing a gentler control than temperature alone. Most of the time, leave it at 0.95 or 1.0.
• Lock your sampling parameters when running evaluations or benchmarks. Variable temperature makes it impossible to measure real quality improvements.
• Document temperature and top_p settings in your code, logs, and experiments. Future you will need to know exactly what settings produced a given result.