Evaluating LLM features: AI & ML lessons from production

Introduction

You ship an LLM feature to production, and initially it feels solid—users aren't complaining, the latency is acceptable, and cost per request is within budget. Then, three weeks later, you make a small change to your system prompt to handle edge cases better, and suddenly your error rate doubles. You don't notice for two days. Evaluating LLM features: AI & ML lessons from production is about building the measurement discipline you need to catch regressions before they compound, and to make confident decisions when you iterate on models, prompts, and retrieval strategies.

The core principle is simple: you cannot improve what you do not measure. In traditional software, we have deterministic tests. In LLM work, we need a different toolkit—one that balances automation with human judgment, and that fits naturally into your development workflow.

Build a test set of real questions

The foundation of any evaluation strategy is a representative test set. This is not a theoretical exercise; it's a concrete collection of questions and scenarios that your users actually encounter.

Start by collecting real queries from your production logs. If you're running a customer support bot, mine your ticket system for the last month of actual customer questions. If you're building a code search feature, gather real search queries from your IDE logs. Aim for 50–200 examples in your initial set, stratified by difficulty or category.

For each example, include context. A customer support bot should preserve the ticket metadata—product, severity, customer segment. A code search feature should include the file types and repository characteristics. This metadata lets you slice your results later and identify where your system struggles.

Here's a concrete structure we've used in production:

# test_set.py
test_cases = [
    {
        "id": "support_001",
        "question": "Why does the checkout button sometimes freeze?",
        "category": "ui_bug",
        "severity": "high",
        "expected_intent": "bug_report",
        "expected_action": "route_to_senior_agent",
        "product": "checkout_v3",
        "created_at": "2025-01-15T10:23:00Z",
    },
    {
        "id": "support_002",
        "question": "Can I use this with my existing Shopify store?",
        "category": "integration",
        "severity": "medium",
        "expected_intent": "compatibility_question",
        "expected_action": "provide_docs_link",
        "product": "api_platform",
        "created_at": "2025-01-16T14:05:00Z",
    },
]

Version control this test set as you would code. When you find a new edge case or misclassification in production, add it to the test set immediately. Over months, this grows into a living artifact that reflects your system's journey.

Score with rubrics or model graders

With your test set in place, you need a way to grade the LLM's output automatically. There are two complementary approaches: rubrics and model-graded evaluation.

Rubrics are explicit scoring rules. For a customer support bot, a rubric might look like:

# rubrics.py
def score_support_response(response, expected_action):
    """
    Returns a score from 0 to 1 and a breakdown.
    """
    score = 0
    breakdown = {}
    
    # Check 1: Does the response mention the correct product?
    if response.get("product") == expected_action.get("product"):
        score += 0.25
        breakdown["product_accuracy"] = True
    else:
        breakdown["product_accuracy"] = False
    
    # Check 2: Is the suggested action appropriate?
    if response.get("action") in expected_action.get("valid_actions", []):
        score += 0.35
        breakdown["action_accuracy"] = True
    else:
        breakdown["action_accuracy"] = False
    
    # Check 3: Is there hallucinated information?
    if not response.get("contains_fabricated_info", False):
        score += 0.25
        breakdown["hallucination_free"] = True
    else:
        breakdown["hallucination_free"] = False
    
    # Check 4: Response latency acceptable?
    if response.get("latency_ms", float('inf')) < 2000:
        score += 0.15
        breakdown["latency_ok"] = True
    else:
        breakdown["latency_ok"] = False
    
    return score, breakdown

Model-graded evaluation delegates scoring to a separate LLM. You ask GPT-4 or Claude to evaluate whether the primary LLM's response is correct, helpful, and factual. This works well when rules are fuzzy—for example, judging whether a generated summary captures the key points. Here's a pattern:

# model_grader.py
def grade_with_claude(test_case, response, grader_model="claude-3-sonnet"):
    """
    Use Claude to grade the response against expected output.
    """
    grading_prompt = f"""
You are evaluating a customer support bot's response.

User question: {test_case['question']}
Expected action: {test_case['expected_action']}
Bot response: {response}

On a scale of 0–10, how well does the bot's response:
1. Correctly identify the issue category?
2. Suggest an appropriate next action?
3. Avoid making up information?

Respond in JSON format:
{{"category_score": , "action_score": , "hallucination_score": , "overall": , "reasoning": ""}}
"""
    
    result = anthropic.Anthropic().messages.create(
        model=grader_model,
        max_tokens=500,
        messages=[{"role": "user", "content": grading_prompt}]
    )
    
    return json.loads(result.content[0].text)

In practice, combine both. Use rubrics for deterministic checks (latency, structured field presence, no hallucinations). Use model graders for semantic quality. This hybrid approach gives you the speed of automation plus the nuance of AI judgment.

Track regressions when prompts change

The moment you make a change—a new system prompt, a different model, a tweak to your retrieval strategy—you must re-score your entire test set and compare against the baseline. This is how you catch regressions before they reach users.

Set up a simple tracking mechanism. We typically use a JSON or CSV file versioned with the codebase:

# eval_results.py
import json
from datetime import datetime

class EvalTracker:
    def __init__(self, results_file="eval_results.json"):
        self.results_file = results_file
        try:
            with open(results_file) as f:
                self.history = json.load(f)
        except FileNotFoundError:
            self.history = []
    
    def log_run(self, version_id, model, prompt_hash, test_set_id, scores):
        """
        Log a full evaluation run.
        scores = {"case_id": float_score, ...}
        """
        run = {
            "timestamp": datetime.utcnow().isoformat(),
            "version_id": version_id,
            "model": model,
            "prompt_hash": prompt_hash,
            "test_set_id": test_set_id,
            "scores": scores,
            "mean_score": sum(scores.values()) / len(scores),
            "min_score": min(scores.values()),
            "max_score": max(scores.values()),
        }
        self.history.append(run)
        self._save()
        return run
    
    def compare_to_baseline(self, run, baseline_version):
        """
        Compare a new run to a baseline version.
        Returns delta in mean score and case-by-case differences.
        """
        baseline = next(
            (r for r in self.history if r["version_id"] == baseline_version),
            None
        )
        if not baseline:
            raise ValueError(f"Baseline {baseline_version} not found")
        
        delta = run["mean_score"] - baseline["mean_score"]
        regressions = {
            case_id: (baseline["scores"][case_id] - run["scores"][case_id])
            for case_id in run["scores"]
            if run["scores"][case_id] < baseline["scores"][case_id]
        }
        
        return {
            "delta_mean": delta,
            "regression_count": len(regressions),
            "regressions": regressions,
        }
    
    def _save(self):
        with open(self.results_file, "w") as f:
            json.dump(self.history, f, indent=2)

Integrate this into your CI/CD pipeline. Before merging a prompt change, automatically run the evaluation suite. If the mean score drops below a threshold (e.g., more than 5% regression), block the merge or flag it for manual review. This prevents silent quality degradation.

Review samples by hand regularly

Automation catches averages and statistical trends, but it misses subtle failure modes. Once a week, spend 30 minutes sampling 5–10 responses from your test set (or from production) and reading them carefully.

Ask yourself:

• Does the bot understand the nuance in the question?
• Is it making reasonable inferences, or is it hallucinating?
• Are there patterns in failures—for example, always struggling with a certain category or tone?
• Would a user find this response helpful, even if the rubric score is high?

Document these findings in a shared doc or issue tracker. If you notice the bot consistently mishandles technical jargon, that's a signal to enrich your test set or adjust your prompt. If you see drift between rubric scores and actual user satisfaction, it's time to refine your rubrics.

This human-in-the-loop review is not a bottleneck; it's a feedback mechanism that keeps your automation honest.

Common mistakes

• Test set too small or unrepresentative. A 10-question test set will miss rare but critical failure modes. Aim for at least 50, and stratify by category and difficulty.
• Rubrics that are too permissive. If your rubric gives a score of 0.8 to a response that contains hallucinated information, you're not catching a real problem. Be strict on factuality.
• No tracking of baseline. Without a clear baseline version, regression detection becomes guesswork. Always tag your models and prompts with a version identifier and keep results timestamped.
• Skipping manual review. Model-graded evaluation can be biased by the grader model itself. Regular manual sampling keeps you grounded in reality.
• Changing too many things at once. If you swap the model, update the prompt, and change your retrieval strategy simultaneously, you cannot isolate what caused a regression. Change one variable at a time.

Key takeaways

• Build a test set of 50–200 real questions representative of your production workload, versioned in your repo.
• Score with a hybrid approach: rubrics for deterministic checks, model graders for semantic quality.
• Track scores over time and block or flag merges that introduce regressions beyond a threshold.
• Spend 30 minutes weekly doing manual spot checks to catch subtle failures and refine your rubrics.
• Version your model, prompt, and evaluation metadata together so you can always trace back why performance changed.
• Treat evaluation as an ongoing practice, not a one-time validation step.