A/B testing ML models in Practical AI & Machine Learning

Introduction

You've trained an ML model, validated it on historical data, and now you're ready to ship. But launching a model directly to all users is risky—backtest performance rarely matches production reality. A/B testing ML models in Practical AI & Machine Learning is how we prove a model actually helps before full rollout, catching performance regressions and user behavior shifts that no offline metric would reveal.

This is Day 77 of the 100-day series, and we're going to walk through the mechanics of running a statistically sound A/B test for ML models, from metric selection through analysis. By the end, you'll know how to design an experiment that protects both users and business metrics.

Define One Primary Metric

Before you randomize a single user, you must declare your primary success metric. This is the one number that determines whether your model wins or loses. Without it, you'll fall into the trap of metric shopping—looking at ten different outcomes and claiming victory on whichever one favors your model.

In e-commerce, your primary metric might be conversion rate or average order value. In a recommendation system, it could be click-through rate on recommended items. In a CRM platform using churn prediction, it might be retention rate at 30 days. The key is choosing something that:

• Directly reflects your business goal
• Moves at a reasonable velocity (you can collect enough samples in days or weeks, not months)
• Isn't gamed by the model or user behavior changes

Let's say you're running a travel platform and you've built a new ML model to rank hotel listings. Your primary metric is booking rate per search session—the fraction of search sessions that result in at least one booking. This metric is tied to revenue, moves fast enough to evaluate in a week, and isn't easily manipulated.

Document this metric upfront in your test plan. Write it down, have your team agree, and lock it before you see any results. This discipline prevents the temptation to pivot mid-experiment.

Randomize Users, Not Requests

This is where many teams go wrong. If you randomize at the request level—sending 50% of ranking requests to the new model and 50% to the old—you'll contaminate your results. The same user might see different hotel orderings on consecutive searches, creating confusion and inconsistent behavior that breaks causal inference.

Instead, randomize users. Pick a stable, user-level identifier (user ID, account ID, session ID, or a hash of IP + user agent for logged-out users) and assign each user to either control (old model) or treatment (new model) at the start of their session or visit. Then every interaction that user has during the test window uses their assigned model.

Here's a practical example in Python:

import hashlib

def get_user_variant(user_id, salt="hotel_ranking_test_v1", split=0.5):
    """
    Deterministic user assignment to A/B test variant.
    Same user always gets the same variant.
    """
    hash_input = f"{user_id}:{salt}".encode()
    hash_value = int(hashlib.md5(hash_input).hexdigest(), 16)
    return "treatment" if (hash_value % 100) < (split * 100) else "control"

def rank_hotels(user_id, search_query, model_registry):
    variant = get_user_variant(user_id)
    model = model_registry.get_model(
        "hotel_ranker_new" if variant == "treatment" else "hotel_ranker_baseline"
    )
    
    ranked_hotels = model.predict_and_rank(search_query)
    
    # Log the variant for later analysis
    log_event(
        event_type="hotels_ranked",
        user_id=user_id,
        variant=variant,
        num_results=len(ranked_hotels)
    )
    
    return ranked_hotels

This approach ensures consistency: a user is either in treatment or control for the entire test window. It also makes analysis cleaner—you're comparing groups of users, not mixing individual requests.

One more consideration: if your system involves batch processing or cached results, make sure your cache key includes the variant. Don't serve cached results from one model to a user in a different variant.

Run Long Enough for Significance

Statistical significance means your observed difference is unlikely to be due to random chance alone. The longer you run your test, the more confident you can be. Too short, and you might make a wrong call; too long, and you're burning time or serving a worse model to half your users.

The required sample size depends on:

• Baseline conversion rate or metric (if your booking rate is 2%, you need more samples than if it's 20%)
• Minimum detectable effect (MDE) (the smallest improvement you care about, e.g., a 5% relative lift)
• Statistical power (typically we aim for 80%—a 20% chance of missing a true effect)
• Significance level (typically 0.05, a 5% chance of a false positive)

Use a power calculator before you launch. Here's a rough formula for continuous metrics:

from scipy.stats import norm

def estimate_sample_size(
    baseline_mean,
    baseline_std,
    effect_size,
    alpha=0.05,
    power=0.80
):
    """
    Estimate per-group sample size for a two-sample t-test.
    effect_size is the absolute difference you want to detect.
    """
    z_alpha = norm.ppf(1 - alpha / 2)
    z_beta = norm.ppf(power)
    
    sample_size = (
        2 * (baseline_std ** 2) * ((z_alpha + z_beta) ** 2) / (effect_size ** 2)
    )
    
    return int(sample_size)

# Example: booking rate baseline is 100 bookings per day in control group
# We want to detect a 10% relative lift (10 more bookings per day)
# Assume daily rate is roughly normally distributed with std=15

n = estimate_sample_size(baseline_mean=100, baseline_std=15, effect_size=10)
print(f"Need approximately {n} days of data per group")
# Output: ~90 days per group (180 days total if alternating)

In practice, if you're splitting traffic 50/50 and your platform does 10,000 searches per day, you'd collect 5,000 searches per variant daily. If you need 90 days per group, run the test for roughly 90 days total with 50/50 split.

One caveat: watch for day-of-week and seasonality effects. Running a test for exactly 7 days might miss weekend behavior. Run for at least two full weeks, ideally two or four weeks. Document your power calculation and planned duration upfront.

Watch Guardrail Metrics Like Latency

Your primary metric is what you're optimizing for, but guardrail metrics protect you from Pyrrhic victories. A new model might boost booking rate by 2% but tank page load time, frustrating users and harming long-term retention. Guardrails catch these hidden costs.

Common guardrails include:

• Latency (p50, p95, p99 response time)
• Error rate (fraction of requests that fail or return invalid results)
• User experience metrics (session abandonment, quick bounces)
• Downstream business metrics (return rate for e-commerce, long-term retention)

Set thresholds for each guardrail before the test. For latency, you might say: "p99 latency must not increase by more than 10ms." If a guardrail is breached, you stop the test and investigate, even if your primary metric looks great.

Here's how you might log and monitor guardrails:

import time
from dataclasses import dataclass

@dataclass
class RequestMetrics:
    user_id: str
    variant: str
    latency_ms: float
    error: bool
    primary_metric: float  # e.g., booking conversion (1 or 0)

def rank_and_measure(user_id, search_query, model_registry):
    start = time.time()
    
    try:
        variant = get_user_variant(user_id)
        model = model_registry.get_model(
            "hotel_ranker_new" if variant == "treatment" else "hotel_ranker_baseline"
        )
        
        ranked_hotels = model.predict_and_rank(search_query)
        error = False
        
    except Exception as e:
        error = True
        ranked_hotels = []
        log_error(f"Ranking failed: {e}")
    
    latency_ms = (time.time() - start) * 1000
    
    # Log for monitoring
    metrics = RequestMetrics(
        user_id=user_id,
        variant=variant,
        latency_ms=latency_ms,
        error=error,
        primary_metric=1.0 if ranked_hotels and user_booked(user_id) else 0.0
    )
    
    send_to_analytics(metrics)
    
    return ranked_hotels

During the test, aggregate these metrics daily and compare control vs. treatment. If treatment latency is creeping upward, flag it immediately. Guardrail violations can force you to roll back even if the primary metric looks promising.

Common Mistakes

Peeking at results mid-test: Checking significance daily tempts you to stop early when p-value looks good, inflating false positive rates. Decide on duration upfront and stick to it. If you must peek, use sequential testing methods and adjust your significance threshold accordingly.

Ignoring time-of-day and day-of-week effects: If you run a test for 3 days Monday–Wednesday, you're missing weekend behavior. Holiday periods and seasonal spikes also matter. Always run at least 14 days, ideally capturing a full week cycle twice.

Not logging the variant with every event: Weeks later, when someone asks "which model was used for user 12345?", you want a reliable audit trail. Log the variant (control or treatment) with every metric you collect. It's cheap insurance.

Conflating statistical significance with practical significance: With massive sample sizes, tiny, irrelevant improvements can be "statistically significant." Define your MDE upfront—the smallest improvement that actually matters to your business. An A/B test proving a 0.1% boost might be statistically sound but economically pointless.

Running multiple models simultaneously without proper multiple-comparison corrections: If you test three new models against a baseline, you triple your false positive risk. Use corrections like Bonferroni, or declare one primary model upfront and test secondary models sequentially in later experiments.

Key Takeaways

• Lock your primary metric before launching the test. Metric shopping leads to false conclusions.
• Randomize users, not requests. Consistent treatment assignment per user ensures valid causal inference.
• Calculate sample size upfront using power analysis. Run for at least 14 days to capture weekly cycles.
• Monitor guardrail metrics (latency, error rate, retention) to catch hidden regressions.
• Document your test plan, variant assignment logic, and decision thresholds. Future you will thank present you.
• Don't peek at results mid-test unless you use sequential testing with adjusted significance levels.
• Statistical significance doesn't mean practical significance. Define an MDE that justifies deployment.