Data leakage: the silent model killer, explained simply

The Problem Nobody Wants to Admit

You've trained a model that achieves 98% accuracy. Your team celebrates. Your stakeholders are thrilled. Then it hits production and drops to 62%. Welcome to data leakage: the silent model killer, explained simply—a mistake so common that almost every team makes it at least once, and many never realize they've made it at all.

Data leakage happens when information from outside the training set accidentally flows into your model during development, inflating performance metrics and creating an illusion of accuracy that vanishes the moment new data arrives. It's not a bug in your code; it's a flaw in how you're thinking about data flow. Let's walk through the most dangerous forms and how to catch them before they derail your project.

Features Computed After the Outcome

This is the most insidious form of leakage because it feels legitimate. You're computing new features from raw data—what could go wrong?

Imagine you're building a fraud detection model for a payment platform. Your outcome variable is whether a transaction was fraudulent. You engineer a feature called days_since_account_flagged, computed from a support ticket system. The problem: that flag was often added after fraud was already detected, so your model learns to recognize fraud from information that wouldn't exist in real time. When you score new transactions, you can't look into the future, and performance tanks.

Another classic example: predicting customer churn. You include a feature like support_tickets_in_last_7_days, which correlates strongly with churn. But those tickets often come because the customer is already leaving. Your model learns the symptom, not the cause. It predicts churn perfectly on historical data—where tickets already exist—but fails on live customers where you'd need to predict churn before they file a ticket.

The fix is temporal consistency: ask yourself, "Would this feature be available at the moment I need to make a prediction?" If not, exclude it. Document the exact timestamp at which features are computed relative to the prediction moment.

# ❌ WRONG: Feature uses information from after the prediction point
def engineer_features(transaction_data, support_tickets):
    features = transaction_data.copy()
    # Support ticket created 2 days after the transaction
    features['recent_support_tickets'] = features.apply(
        lambda row: sum(support_tickets[
            (support_tickets['user_id'] == row['user_id']) &
            (support_tickets['created_at'] > row['transaction_time'])  # LEAK!
        ]), axis=1
    )
    return features

# ✓ RIGHT: Feature uses only information available at prediction time
def engineer_features(transaction_data, support_tickets, cutoff_days=30):
    features = transaction_data.copy()
    # Tickets from before the transaction
    features['prior_support_tickets'] = features.apply(
        lambda row: sum(support_tickets[
            (support_tickets['user_id'] == row['user_id']) &
            (support_tickets['created_at'] < row['transaction_time']) &
            (support_tickets['created_at'] > row['transaction_time'] - timedelta(days=cutoff_days))
        ]), axis=1
    )
    return features

Scaling Before the Train-Test Split

Normalization and scaling are standard preprocessing steps. They make sense. But timing matters more than you'd think.

When you compute mean and standard deviation across your entire dataset before splitting into train and test, your test set's statistics are baked into how your training features are scaled. The model sees test data statistics during training through the scaling parameters. This is particularly dangerous with StandardScaler or MinMaxScaler applied to the full dataset upfront.

Here's why it hurts: imagine you're predicting house prices and 5% of your data contains luxury penthouses worth $10M. If you scale using the global max before splitting, your training data's features are scaled relative to those luxury outliers. When your model trains, it learns patterns in a scaled space informed by data it shouldn't know about. Your test performance looks great. In production, when you encounter a new neighborhood without those extreme values, the scaling behaves differently and predictions drift.

# ❌ WRONG: Fit scaler on entire dataset, then split
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

X, y = load_data()
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)  # LEAK: fit on all data including test

X_train, X_test, y_train, y_test = train_test_split(X_scaled, y)
model = LogisticRegression()
model.fit(X_train, y_train)
print(model.score(X_test, y_test))  # Inflated score

# ✓ RIGHT: Split first, then fit scaler only on training data
X, y = load_data()
X_train, X_test, y_train, y_test = train_test_split(X, y)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # Fit only on train
X_test_scaled = scaler.transform(X_test)  # Transform test with train's statistics

model = LogisticRegression()
model.fit(X_train_scaled, y_train)
print(model.score(X_test_scaled, y_test))  # Honest score

The same principle applies to other preprocessing: PCA, imputation of missing values, outlier removal, feature selection. Always fit these on training data only, then apply to test and production.

Duplicates Across Train and Test

If the same sample appears in both your training and test sets, your model has literally memorized part of the correct answer. This happens more often than you'd expect, especially in real-world messy datasets.

Consider a ride-hailing platform. Customers take rides repeatedly. You're predicting ride duration. Your raw data is 500,000 ride records. When you do a simple random split 80/20, there's a high probability that some of the same routes—or even the same customer, same time of day, same location—appear in both sets. Your model learns "this specific route takes 23 minutes" during training and "recognizes" it during testing, inflating your metrics.

In e-commerce recommendation systems, duplicates are especially common. If your dataset is user-item interactions and the same user-product pair appears multiple times, a naive split will leak information about what that user liked to the test set.

# ❌ WRONG: Simple random split can have duplicates
from sklearn.model_selection import train_test_split

data = load_ride_data()  # 500k rides
X_train, X_test, y_train, y_test = train_test_split(
    data.drop('duration', axis=1),
    data['duration'],
    test_size=0.2
)

# Some customer-route-time combinations appear in both sets
print("Duplicates in train:", X_train.duplicated().sum())
print("Duplicates in test:", X_test.duplicated().sum())
print("Overlap:", len(set(X_train.index) & set(X_test.index)))  # Likely > 0

# ✓ RIGHT: Group by customer or entity, split at the entity level
from sklearn.model_selection import GroupShuffleSplit

data = load_ride_data()
splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)

for train_idx, test_idx in splitter.split(
    data,
    groups=data['customer_id']  # Split by customer, not by ride
):
    X_train, X_test = data.iloc[train_idx], data.iloc[test_idx]
    y_train, y_test = data['duration'].iloc[train_idx], data['duration'].iloc[test_idx]

# No customer appears in both sets
print("Overlap:", len(set(X_train['customer_id']) & set(X_test['customer_id'])))  # 0

If you have explicit duplicate rows in your dataset, catch them before splitting. If you have logical duplicates (same customer, same context), use stratified or grouped splitting strategies.

Too-Good Scores Are a Warning Sign

One of the best early indicators of data leakage is suspiciously high performance. A model achieving 95%+ accuracy should trigger alarm bells, not celebration—at least until you've verified why.

In our experience, models in production typically achieve 70–85% accuracy on realistic problems with well-engineered features. If your first iteration hits 95%, you're either:

• Solving an unusually easy problem (rare)
• Have leaked information into your training process (common)
• Overfitting on a small dataset (also common)

A practical defense: establish a baseline with a simple model first. If you're predicting fraud, train a logistic regression on a few obvious features and see what baseline accuracy you get. If your fancy deep learning model only beats that baseline by 2%, maybe the problem is inherently easy. If it beats it by 40%, investigate what's different between your complex model and the baseline. The gap often points to where leakage is happening.

Another sanity check: cross-validation. If your cross-validation scores are dramatically different from your hold-out test score, something's wrong with how you're splitting or preprocessing.

Common Mistakes

• Using the full dataset to select features. Feature selection should happen on training data only, or you're encoding test set information into which features to use.
• Removing outliers globally before splitting. If you remove outliers from the full dataset, you're removing them based on information from the test set.
• Imputing missing values with global statistics. Compute mean/median/mode from training data only, then apply to test.
• Treating time-series data as unordered. In time-series problems, always split by time. Never randomly shuffle and split.
• Including metadata that's only available after the outcome. Timestamps, flags, or tags that are added after the event you're predicting are almost always leaks.
• Relying on a single train-test split. Use cross-validation to catch leakage you might miss in one random split.

Key Takeaways

• Data leakage inflates training metrics while production performance crashes. It's invisible until deployment.
• Ask yourself: "Would this feature or preprocessing step be available at the moment I need to make a prediction?" If not, it's a leak.
• Always split your data before fitting preprocessing steps like scaling, imputation, or feature selection.
• Catch logical duplicates by using grouped or stratified splitting when your data has repeated entities.
• Suspiciously high accuracy (95%+) is a warning sign. Establish a simple baseline and investigate large gaps.
• Use cross-validation to detect leakage patterns you might miss in a single hold-out split.
• Document temporal relationships in your features. When is each piece of information available relative to the prediction moment?