Train, validation and test sets: lessons from production

Opening hook

We've all been there: a model performs beautifully in development, then tanks in production. The culprit is usually not the algorithm—it's how we evaluated it. Train, validation and test sets: lessons from production reveals why splitting your data into three distinct parts isn't bureaucracy; it's survival. Getting this right means the difference between shipping a model you can trust and shipping one that silently degrades.

The core problem: data leakage and overfitting

When we train a machine learning model, we're optimizing weights to minimize error on whatever data we show it. If we then measure performance on the same data, we're not measuring generalization—we're measuring memorization. The model has seen these examples before, and it cheats.

In production, your model encounters data it has never seen. That's where reality hits. The three-way split exists to simulate that future reality as honestly as possible, at each stage of development:

• Training set: used to fit model weights.
• Validation set: used to tune hyperparameters and select models.
• Test set: held back completely, opened only once, to get an honest final estimate.

Many teams collapse this into two sets or reuse the validation set for testing. We've seen that end in production incidents.

Train: fit the model

The training set is where your model learns. You pass examples through the network, compute loss, backpropagate, and update weights. This set should be large enough to cover the problem space, but you don't need to overthink size—modern deep learning often improves with more training data, but the relationship isn't linear.

In a real project we worked on, a recommendation engine for an e-commerce platform used 80% of historical user interactions as the training set. That amounted to roughly 2.5 million user-product pairs. The model saw a diverse range of user behaviors, seasonal patterns, and product categories.

One key discipline: never look at validation or test performance while tweaking training hyperparameters like learning rate or batch size. Those changes affect training dynamics, and if you stare at validation loss while turning knobs, you're implicitly tuning on the validation set. Train in isolation. Log metrics. Move on.

import numpy as np
from sklearn.model_selection import train_test_split

# Assume X, y are your features and labels
X_train, X_temp, y_train, y_temp = train_test_split(
    X, y, test_size=0.3, random_state=42
)

# Split the remaining 30% into validation (15%) and test (15%)
X_val, X_test, y_val, y_test = train_test_split(
    X_temp, y_temp, test_size=0.5, random_state=42
)

print(f"Train: {len(X_train)} | Validation: {len(X_val)} | Test: {len(X_test)}")

A typical split is 70% train, 15% validation, 15% test. In larger datasets, train can be higher; in smaller datasets, you might need K-fold cross-validation on the training set to make better use of limited data.

Validation: tune hyperparameters

Once training stabilizes, you evaluate on the validation set. This is where you experiment. You might try different learning rates, regularization strengths, model depths, or architectural choices. You watch validation loss and accuracy, pick the best configuration, and move forward.

The validation set is your development feedback loop. It's fast, it's yours to probe, and it answers: "Which model variant should I ship?" Think of it as your private testing ground before the final exam.

In the e-commerce recommendation project, the validation set (about 600k user-product pairs) let us quickly test whether adding temporal features improved ranking. We tried three versions:

• Baseline (no temporal features)
• Recency weighted (recent interactions count more)
• Seasonal embeddings (separate embeddings per quarter)

Validation metrics favored seasonal embeddings. We selected that version and locked it in. Then we opened the test set once.

from sklearn.linear_model import LogisticRegression

# Hyperparameter search on validation set
best_score = 0
best_model = None

for C in [0.001, 0.01, 0.1, 1, 10]:
    model = LogisticRegression(C=C, max_iter=1000)
    model.fit(X_train, y_train)
    val_score = model.score(X_val, y_val)
    
    if val_score > best_score:
        best_score = val_score
        best_model = model
    
    print(f"C={C}: validation score = {val_score:.4f}")

print(f"\nBest C: {best_model.C}, Best validation score: {best_score:.4f}")

The critical rule: never tune on the test set. Once you touch test data to make decisions, it's contaminated. You're back to measuring memorization, not generalization.

Test: one final honest estimate

The test set is sacred. You open it exactly once, after all development is complete. You run your final model on it, record the numbers, and that's your expected production performance. This is the number you report to stakeholders. This is the number you use to decide "go or no-go."

For the recommendation engine, test set performance (about 600k held-out examples) showed a 12% improvement in ranking precision over the baseline—exactly what validation had predicted. We shipped with confidence.

If test performance surprises you—better or worse than validation—that's valuable signal. Better: great, but don't trust it; run production and verify. Worse: dig into why. Is there a data distribution shift? Did you accidentally introduce a bug? Did the validation set have a lucky cluster of easy examples?

from sklearn.metrics import accuracy_score, precision_score, recall_score, f1_score

# Evaluate best model on test set (once, at the end)
y_test_pred = best_model.predict(X_test)

print("Test Set Results:")
print(f"Accuracy:  {accuracy_score(y_test, y_test_pred):.4f}")
print(f"Precision: {precision_score(y_test, y_test_pred):.4f}")
print(f"Recall:    {recall_score(y_test, y_test_pred):.4f}")
print(f"F1:        {f1_score(y_test, y_test_pred):.4f}")

Keep test set results and let them inform monitoring. If production metrics drift below test performance, something in the real world has changed. That's your signal to retrain or investigate data quality.

Never tune on the test set

This deserves its own section because we've seen it happen in production code, and it's a silent killer.

Tuning on the test set means using test metrics to decide which model to ship. You run three models, check their test scores, pick the highest, and deploy it. You've now committed your model selection to test data. Future performance will almost certainly be worse, because you've optimized for noise in that particular test set.

The same applies to hyperparameter sweeps. If you run a grid search over 50 hyperparameter combinations and pick the best one based on test accuracy, you're selecting for the combination that got lucky on that test set. The validation set exists to absorb this risk. Use it.

The golden rule: Validation tunes, test confirms. Never both.

Common mistakes

• Using only train/test. You'll tune hyperparameters on test data without realizing it, and validation artifacts become production surprises.
• Stratified splits ignored. If your target class is imbalanced, random splits can distribute the minority class unevenly across train, validation, and test. Use stratify=y in scikit-learn or group-aware splits for time series.
• Time leakage in time-series models. Shuffling data before split can leak future information into the training set. For time series, always split chronologically: train on past, validate/test on future.
• Reusing validation set across experiments. Each experiment should ideally have its own validation split to avoid implicit tuning on the same examples.
• Opening test set "just to check." Once you peek, the seal is broken. If you do, record those results separately and keep one final hold-out for true production validation.

Key takeaways

• Split data into three disjoint sets: train (fit), validation (tune), test (confirm).
• Train set is where the model learns weights; it's yours to scale and iterate on.
• Validation set is where you experiment with hyperparameters and select models; tune aggressively here.
• Test set is sealed until the end; it gives you an honest estimate of production performance.
• Never tune on the test set. The moment you use test metrics to make model decisions, test data is contaminated and your final estimate is biased.
• In time-series problems, respect chronological order; split by time, not randomly.
• Monitor production carefully. If real-world metrics drift below test performance, investigate and retrain.