Common ML mistakes in production for AI & ML developers

Introduction

We've all shipped a model that looked perfect in notebooks. Then it hits production and performance tanks. Common ML mistakes in production for AI & ML developers are rarely about the algorithm—they're about the gap between what we build and what the system actually encounters. After years of deploying models across e-commerce, trading, and SaaS platforms, we've learned these lessons the hard way so you don't have to.

This article walks through four critical failures we see repeatedly: training on data the model will not see in production, shipping without a baseline to compare against, ignoring drift, and deploying without monitoring. Each one has taken down production systems. Each one is preventable.

Training on data the model will not see in production

The most common mistake is straightforward: your training data doesn't represent what the model encounters once live.

Consider a ride-hailing platform. Your team builds a demand forecasting model using six months of historical data from major cities. The model trains on a specific set of routes, times of day, and user patterns. It validates beautifully on your test set. Then you deploy it, and suddenly it's predicting negative demand or missing evening rush hour because the test data didn't include a major event that draws traffic differently. Or worse: you trained on summer data, deploy in winter, and weather patterns the model never saw dominate performance.

The root cause is data leakage in reverse. You're not leaking target information into features—you're leaking assumptions about the operational environment into your training pipeline. Production sees users, events, geographies, and temporal patterns your training set never captured.

Real scenario: an e-commerce platform built a conversion prediction model on desktop users. Mobile traffic was rising but underrepresented in training data. When the model went live, mobile predictions were poor because the feature distributions (session length, click patterns, device type features) were fundamentally different. The team had to retrain with a stratified sample and implement online retraining for new traffic patterns.

Fix this by:

• Collecting production data before you train. Even a small sample of real traffic tells you what your model will actually face.
• Using stratified train-test splits across key dimensions: geography, device type, user cohort, time of year.
• Building a feature store that mirrors production schemas and data distributions.
• Testing the model on holdout data that's temporally distant from training—not just randomly shuffled.

# Bad: random train-test split
from sklearn.model_selection import train_test_split

X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)
model.fit(X_train, y_train)

# Good: temporal split with stratification
train_cutoff = '2024-01-15'
test_cutoff = '2024-02-01'

X_train = X[X.timestamp < train_cutoff]
X_test = X[(X.timestamp >= test_cutoff) & (X.timestamp < test_cutoff + timedelta(days=7))]

# Stratify by production dimensions
from sklearn.model_selection import StratifiedShuffleSplit

sss = StratifiedShuffleSplit(
    n_splits=1,
    test_size=0.2,
    random_state=42
)
for train_idx, test_idx in sss.split(X, X['device_type']):
    X_train, X_test = X.iloc[train_idx], X.iloc[test_idx]
    y_train, y_test = y.iloc[train_idx], y.iloc[test_idx]

No baseline to compare against

You've trained a model. It achieves 85% accuracy. Great, right? Without a baseline, you have no idea if that's good.

A baseline is what you're replacing. It might be a rule-based system, a previous model version, or the simplest possible predictor. On a classification task, baseline could be "always predict the majority class." On a regression task, it could be "predict the mean." On a ranking task, it could be "use the existing heuristic." If your fancy model doesn't beat the baseline, it shouldn't go to production.

Real example: a SaaS platform built a customer churn prediction model that achieved 90% accuracy. They were ready to deploy. One engineer asked: "What does just predicting everyone will churn give us?" Answer: 91% accuracy, because only 9% of customers actually churn. The model was worse than guessing.

Baselines catch this immediately. They also let you measure actual business impact. A 2% accuracy improvement might sound good until you realize the previous system (human review, simple rules, random guessing) already achieves the outcome your team cares about.

Set a baseline before training:

• Statistical baseline: Majority class, mean value, or stratified random prediction.
• Operational baseline: Whatever system you're replacing—the existing rule set, previous model version, or manual process.
• Domain baseline: What a domain expert would predict with zero data (for time-series, often "last value carried forward").

from sklearn.metrics import accuracy_score, f1_score, mean_absolute_error
import numpy as np

# Baseline: always predict majority class
majority_class = y_test.mode()[0]
baseline_pred = np.full(len(y_test), majority_class)
baseline_accuracy = accuracy_score(y_test, baseline_pred)

# Your model
model_pred = model.predict(X_test)
model_accuracy = accuracy_score(y_test, model_pred)

# Compare
print(f"Baseline accuracy: {baseline_accuracy:.3f}")
print(f"Model accuracy: {model_accuracy:.3f}")
print(f"Improvement: {(model_accuracy - baseline_accuracy):.3f}")

if model_accuracy < baseline_accuracy:
    print("WARNING: Model does not beat baseline. Do not deploy.")

Ignoring drift

Your model performs well on day one. By week three, it's degrading. By month two, it's useless. This is drift, and it's one of the most common reasons production models fail silently.

Drift comes in two flavors. Concept drift means the relationship between features and target has changed. A credit risk model trained on pre-pandemic data behaves differently in recession. A travel demand model trained during normal times fails during lockdown. Data drift (or covariate shift) means the distribution of input features has changed, but the relationship hasn't necessarily changed. Your model trained on users aged 25–45; suddenly you're getting 18–22 year-old users with different behavior patterns.

The mistake isn't that drift exists—it will. The mistake is not monitoring for it and not retraining when it occurs.

Real scenario: a trading platform built a price prediction model on three years of stable market data. It deployed and ran for eight months with no monitoring. When volatility spiked during a market event, predictions became useless. The team discovered the failure only when traders complained. They lost weeks of performance and trust because they never tracked whether feature distributions were changing.

Detect and respond to drift:

• Log feature distributions and target distributions in production, continuously.
• Use statistical tests (Kolmogorov-Smirnov, Wasserstein distance) to detect when distributions diverge from training.
• Track model performance on recent data. If it's declining and you haven't changed the ground truth labeling, drift is likely.
• Set automated triggers to retrain or alert when drift is detected.

from scipy.stats import ks_2samp
from scipy.spatial.distance import jensenshannon
import pandas as pd
import numpy as np

# Training set statistics
train_feature_mean = X_train['user_age'].mean()
train_feature_std = X_train['user_age'].std()

# Production batch (e.g., daily or weekly)
prod_batch = X_prod_today['user_age']

# Kolmogorov-Smirnov test
ks_statistic, p_value = ks_2samp(X_train['user_age'], prod_batch)
print(f"KS statistic: {ks_statistic:.4f}, p-value: {p_value:.4f}")

if p_value < 0.05:
    print("DRIFT DETECTED: Feature distribution has changed significantly")

# Track model performance over time
recent_performance = evaluate_on_recent_data(model, X_prod_last_7_days, y_prod_last_7_days)
training_performance = model.evaluate(X_test, y_test)

if recent_performance['accuracy'] < training_performance['accuracy'] - 0.05:
    print("Performance degradation detected. Triggering retrain.")
    retrain_model()

Shipping without monitoring

You've validated the model, it's passed review, and it's in production. Now you move on to the next project. This is the moment disaster is most likely.

Without monitoring, you won't know if the model is silently failing. It won't error—it will just make wrong predictions, day after day, while your team is unaware. By the time business metrics reveal the problem weeks later, you've already impacted customers, revenue, or user experience.

Production monitoring for ML isn't optional. It's as critical as monitoring your database or API gateway.

Real example: a CRM platform deployed a lead scoring model. No one set up monitoring. The model shipped and ran for six weeks. During a data pipeline maintenance window, an upstream data source changed schema. The feature engineering code silently failed, filling missing values with defaults. The model continued to score every lead as medium priority. Sales team efficiency tanked. Business users didn't know why. The outage went undetected for three days because no one was watching.

What to monitor:

• Input statistics: Feature distributions, missing value rates, out-of-range values.
• Model outputs: Prediction distributions, confidence scores, error rates.
• Business metrics: Conversion rate, revenue, user satisfaction—whatever changed because of the model.
• Latency and errors: Response time, inference failures, timeouts.

# Example monitoring setup
import logging
from datetime import datetime, timedelta

logger = logging.getLogger('ml_monitoring')

def predict_with_monitoring(model, X_batch, batch_id):
    # Input validation
    null_rate = X_batch.isnull().sum().sum() / (len(X_batch) * len(X_batch.columns))
    if null_rate > 0.1:
        logger.warning(f"High null rate in batch {batch_id}: {null_rate:.2%}")
    
    # Prediction
    start_time = datetime.now()
    predictions = model.predict(X_batch)
    inference_time = (datetime.now() - start_time).total_seconds()
    
    # Output validation
    pred_mean = predictions.mean()
    pred_std = predictions.std()
    
    # Log metrics
    logger.info(f"Batch {batch_id}: inference_time={inference_time:.3f}s, "
                f"pred_mean={pred_mean:.4f}, pred_std={pred_std:.4f}")
    
    # Alert on anomalies
    if inference_time > 1.0:
        logger.error(f"Inference latency spike in batch {batch_id}")
    
    if pred_std < 0.01:
        logger.warning(f"Prediction variance too low in batch {batch_id}. "
                       f"Model may have collapsed.")
    
    return predictions

# Set up dashboard queries
MONITORING_QUERIES = {
    'daily_accuracy': """
        SELECT date, accuracy, recall, precision 
        FROM model_performance 
        WHERE model_id = 'lead_scorer_v2' 
        ORDER BY date DESC LIMIT 30
    """,
    'feature_drift': """
        SELECT feature_name, ks_statistic, p_value 
        FROM feature_drift_detection 
        WHERE model_id = 'lead_scorer_v2' 
        AND timestamp > now() - interval '7 days'
        AND p_value < 0.05
    """,
}

Common mistakes

Beyond the four main issues, we see these patterns repeatedly:

• Forgetting that training is not deployment. Your local machine, Jupyter notebook, and training environment are not production. Different Python versions, library versions, hardware, and data distributions will trip you up.
• Using live labels for validation. If you wait for ground truth labels to arrive, you can't validate early. Build a proxy or early indicator first, even if imperfect.
• Over-engineering without operational understanding. A simple linear model that you can monitor and retrain beats a complex ensemble you don't fully understand and can't debug in production.
• Assuming historical performance predicts future performance. Your test set accuracy doesn't guarantee anything about next week. Only continuous production monitoring does.

Key takeaways

• Train on data representative of production. Temporal splits and stratification matter more than raw accuracy numbers.
• Always establish a baseline before training. If your model doesn't beat what it's replacing, stop.
• Build drift detection into your deployment plan from day one. Monitor feature distributions, target distributions, and model performance continuously.
• Instrument your model like you instrument your database. Log predictions, latencies, input statistics, and business outcomes. Set alerts for anomalies.
• Plan for retraining. Drift is inevitable. Have a process to detect it, trigger retraining, and deploy updates without manual intervention.