Imbalanced classification: practical tips and trade-offs

The problem: rare events, broken metrics

You've built a fraud detection model that's 98% accurate. Your team celebrates. Then in production, it flags almost nothing as fraud, and real fraud slips through. You're experiencing imbalanced classification—when the positive class (fraud, churn, rare disease) is vastly outnumbered by the negative class (legitimate transactions, retained customers, healthy patients). In this scenario, a naive classifier could achieve 98% accuracy by simply predicting "not fraud" for everything.

Imbalanced classification is one of the most common real-world machine learning problems, especially in e-commerce, security, and healthcare. The challenge isn't just building a model; it's building one that catches the rare cases you actually care about while keeping false positives manageable. This article walks through imbalanced classification: practical tips and trade-offs that we've applied across payment systems, risk scoring, and anomaly detection in production.

Use the right metric: Precision, recall, and PR-AUC instead of accuracy

Accuracy is a trap. When your dataset is 99% negative examples, a model that predicts everything as negative achieves 99% accuracy—useless. Instead, focus on metrics that care about the minority class:

• Precision: Of the cases you flagged as positive, how many actually were? (Reduces false alarms.)
• Recall: Of all the true positives, how many did you catch? (Reduces missed cases.)
• F1-score: The harmonic mean of precision and recall. Useful when you want balance, but both matter.
• PR-AUC: Area under the precision-recall curve. More informative than ROC-AUC for imbalanced data because ROC-AUC can be artificially high when negatives dominate.

In a fraud detection pipeline with 0.5% fraud rate, a model with 95% recall (catches 95% of fraud) and 70% precision (30% false positive rate) is often far more valuable than one with 99% accuracy and 30% recall. Your business cost determines which trade-off you prefer—and we'll tune that later.

from sklearn.metrics import precision_recall_curve, auc, f1_score, roc_auc_score
from sklearn.metrics import precision_recall_display
import matplotlib.pyplot as plt

# Assume y_true and y_pred_proba are your labels and model probabilities
precision, recall, thresholds = precision_recall_curve(y_true, y_pred_proba)
pr_auc = auc(recall, precision)

print(f"PR-AUC: {pr_auc:.3f}")
print(f"ROC-AUC: {roc_auc_score(y_true, y_pred_proba):.3f}")

# Plot the precision-recall curve
precision_recall_display(precision, recall).plot()
plt.show()

Start every imbalanced project by settling on a metric with your product team. Then track it obsessively through validation and production.

Class weights: the easiest fix

Most classifiers have a class_weight parameter. Setting it to 'balanced' tells the model to penalize mistakes on the minority class more heavily. It's often the first thing we try, and it works surprisingly well.

from sklearn.ensemble import RandomForestClassifier
from sklearn.linear_model import LogisticRegression

# Random Forest with balanced class weights
rf = RandomForestClassifier(
    class_weight='balanced',
    n_estimators=100,
    random_state=42
)

# Logistic Regression with balanced class weights
lr = LogisticRegression(
    class_weight='balanced',
    max_iter=1000,
    random_state=42
)

rf.fit(X_train, y_train)
lr.fit(X_train, y_train)

Behind the scenes, class_weight='balanced' scales weights inversely to class frequency. If you have 1,000 negatives and 10 positives, the positive class gets weight 100, so misclassifying a positive example costs 100 times as much. This forces the model to pay attention to the minority class.

The trade-off: balanced class weights might hurt overall accuracy slightly, but that's fine—accuracy wasn't your metric anyway. For many teams, this alone lifts recall significantly without needing complex resampling.

Resampling: oversample or undersample, but only on training data

When class weights aren't enough, resampling helps. You can oversample the minority class (duplicate or generate synthetic examples) or undersample the majority class (randomly remove examples). The key rule: always resample only the training set, never the test or validation sets. Resampling the test set inflates your metrics.

Oversampling with SMOTE

Random duplication of minority examples risks overfitting. SMOTE (Synthetic Minority Oversampling Technique) generates new synthetic samples by interpolating between nearest neighbors of minority examples. It's more robust:

from imblearn.over_sampling import SMOTE
from sklearn.model_selection import train_test_split

# Split first, then apply SMOTE only to training data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

# Apply SMOTE to training data only
smote = SMOTE(random_state=42)
X_train_resampled, y_train_resampled = smote.fit_resample(X_train, y_train)

print(f"Original training distribution: {sum(y_train)} positives out of {len(y_train)}")
print(f"After SMOTE: {sum(y_train_resampled)} positives out of {len(y_train_resampled)}")

# Train your model on resampled data
model.fit(X_train_resampled, y_train_resampled)

# Evaluate on original test set
y_pred = model.predict(X_test)
print(f"Precision: {precision_score(y_test, y_pred):.3f}")
print(f"Recall: {recall_score(y_test, y_pred):.3f}")

Undersampling

If your dataset is enormous and you're running into memory or time constraints, undersampling the majority class can help. The trade-off: you're throwing away data, which can hurt generalization. Use stratified undersampling to keep the same feature distributions:

from imblearn.under_sampling import RandomUnderSampler

# Resample training data
undersample = RandomUnderSampler(random_state=42)
X_train_resampled, y_train_resampled = undersample.fit_resample(X_train, y_train)

In production systems, we often use combined resampling: light oversampling of the minority class plus light undersampling of the majority. This keeps both signal and computational cost manageable.

Tune the threshold: choose the cut-off that matches business costs

Most classifiers output probabilities, not hard predictions. By default, we use a threshold of 0.5: if predict_proba(x) >= 0.5, predict positive. For imbalanced data, 0.5 is usually wrong.

A fraud detector that needs to catch 95% of fraud might use a threshold of 0.3. A medical screening model that wants to avoid false alarms might use 0.7. The right threshold depends on your business costs: how expensive is a false positive versus a missed positive?

from sklearn.metrics import precision_recall_curve, f1_score
import numpy as np

# Get predicted probabilities
y_pred_proba = model.predict_proba(X_test)[:, 1]

# Compute precision and recall for all thresholds
precision, recall, thresholds = precision_recall_curve(y_test, y_pred_proba)

# Find the threshold that maximizes F1 (or another metric)
f1_scores = 2 * (precision * recall) / (precision + recall + 1e-10)
best_idx = np.argmax(f1_scores)
best_threshold = thresholds[best_idx]

print(f"Best threshold (F1-based): {best_threshold:.3f}")
print(f"Precision: {precision[best_idx]:.3f}, Recall: {recall[best_idx]:.3f}")

# Or, choose based on business logic
# If false positive cost is 10x false negative cost:
# threshold = find_threshold_by_cost(precision, recall, thresholds, fp_cost=10, fn_cost=1)

# Apply the threshold
y_pred_tuned = (y_pred_proba >= best_threshold).astype(int)

This is often where machine learning meets product. A threshold tuned to F1 might not match what your operations team can handle. Have that conversation early, and build the threshold as a configurable parameter in your pipeline, not a hard-coded constant.

Putting it together: a minimal pipeline

In production, we combine these tactics:

from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from imblearn.over_sampling import SMOTE
from imblearn.pipeline import Pipeline as ImbPipeline
from sklearn.model_selection import train_test_split, cross_validate
from sklearn.metrics import precision_recall_curve
import numpy as np

# Split data
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Build imbalanced-aware pipeline
pipeline = ImbPipeline([
    ('scaler', StandardScaler()),
    ('smote', SMOTE(random_state=42)),
    ('model', RandomForestClassifier(
        class_weight='balanced',
        n_estimators=100,
        random_state=42
    ))
])

# Train
pipeline.fit(X_train, y_train)

# Get probabilities and tune threshold
y_pred_proba = pipeline.predict_proba(X_test)[:, 1]
precision, recall, thresholds = precision_recall_curve(y_test, y_pred_proba)
f1_scores = 2 * (precision * recall) / (precision + recall + 1e-10)
best_threshold = thresholds[np.argmax(f1_scores)]

# Evaluate with tuned threshold
y_pred_tuned = (y_pred_proba >= best_threshold).astype(int)
print(f"Final recall: {recall_score(y_test, y_pred_tuned):.3f}")
print(f"Final precision: {precision_score(y_test, y_pred_tuned):.3f}")

Common mistakes

• Resampling before splitting: This leaks information and inflates metrics. Always split first, then resample only the training set.
• Optimizing for accuracy: On imbalanced data, accuracy is nearly meaningless. Your test metrics will look great and production will fail. Use precision, recall, or PR-AUC.
• Forgetting stratification: When splitting an imbalanced dataset, use stratify=y in train_test_split to ensure both sets have the same class distribution.
• Ignoring the threshold: A well-tuned model with a poorly chosen threshold will underperform. Threshold tuning is as important as model selection.
• Oversampling too aggressively: Inflating the minority class to 50% of your training data often causes overfitting. Gentle oversampling (30-40% minority) often works better.

Key takeaways

• Accuracy is useless for imbalanced classification. Use precision, recall, F1-score, or PR-AUC instead, and agree with your team which metric matters most.
• Start with class_weight='balanced'. It's simple, works well, and is your first tool before reaching for complex resampling.
• If you resample, use SMOTE for oversampling and always resample only the training set, never test or validation data.
• Tune your decision threshold to match business costs. A 0.5 default is almost never right for imbalanced problems.
• Use stratified splits and monitor PR-AUC in cross-validation to catch overfitting early.