Bias and fairness in ML: AI & ML lessons from production

Bias and fairness in ML: AI & ML lessons from production

We've all seen the headlines: a recruiting model that discriminates against women, a lending algorithm that denies loans to minorities, a content recommendation system that radicalizes users. These aren't hypothetical problems—they're real failures in production systems that affected real people. The uncomfortable truth we've learned across countless projects is that models learn the biases in their data, and those biases don't disappear just because we wrapped them in machine learning.

Fairness in machine learning isn't a nice-to-have or a compliance checkbox. It's a fundamental requirement for building systems that work for everyone, and ignoring it will cost you in user trust, legal exposure, and engineering credibility. Let's walk through what we've learned about bias and fairness in ML from shipping models to production.

Check performance across user groups

The first and most critical step is measuring how your model actually performs for different segments of your user base. A model that achieves 95% accuracy overall might be 99% accurate for one demographic and 80% accurate for another. You won't know unless you deliberately slice your metrics.

In our experience, we built a fraud detection model for a payments platform. The overall false positive rate looked acceptable at 2%, but when we broke it down by country, we discovered the model was flagging 15% of transactions in emerging markets as fraudulent while only flagging 1% in developed markets. The model had learned from historical data that transactions from certain regions were riskier—a pattern that reflected past operational decisions, not actual risk.

Here's how we approach this:

import pandas as pd
from sklearn.metrics import precision_recall_fscore_support

# Load predictions and ground truth
predictions = pd.read_csv('model_predictions.csv')

# Define your demographic groups
sensitive_attributes = ['country', 'user_segment', 'income_bracket']

# Calculate metrics per group
for attr in sensitive_attributes:
    for group in predictions[attr].unique():
        group_data = predictions[predictions[attr] == group]
        
        precision, recall, f1, support = precision_recall_fscore_support(
            group_data['true_label'],
            group_data['predicted_label'],
            average='weighted'
        )
        
        print(f"{attr}={group}: precision={precision:.3f}, recall={recall:.3f}, F1={f1:.3f}, n={support}")

The key is consistency. Set acceptable thresholds for performance variance across groups. A common rule of thumb: if any group's key metrics fall below 80% of the average, investigate. Better yet, use fairness metrics like demographic parity (similar positive prediction rates) or equalized odds (similar false positive and false negative rates across groups).

Document these breakdowns in your model card or system documentation. When stakeholders ask "does this model work fairly?", you need real numbers, not assumptions.

Remove proxy features for sensitive attributes

You can't just exclude the sensitive column and call it a day. Sensitive attributes have proxies—features that correlate strongly enough to reconstruct the original bias. In lending, ZIP code proxies for race. In hiring, name or graduation year can proxy for gender or age. In healthcare, socioeconomic status proxies for race.

We worked on a customer churn prediction model for a ride-hailing platform. We removed "user_country" thinking that would handle fairness concerns. But the model still discriminated because it learned from neighborhood patterns, average ride fare, and language preferences—all of which correlated with country and allowed the model to reconstruct country-level bias implicitly.

The fix isn't just removing columns; it's understanding which features are proxies and deciding consciously whether to keep them:

from sklearn.preprocessing import StandardScaler
import numpy as np

def calculate_proxy_correlation(features_df, sensitive_attr):
    """Identify which features correlate strongly with sensitive attributes."""
    scaler = StandardScaler()
    features_scaled = scaler.fit_transform(features_df.drop(columns=[sensitive_attr]))
    sensitive_scaled = scaler.fit_transform(features_df[[sensitive_attr]])
    
    correlations = np.abs(np.corrcoef(features_scaled.T, sensitive_scaled.T)[:-1, -1])
    
    proxy_features = []
    for i, corr in enumerate(correlations):
        if corr > 0.6:  # arbitrary but practical threshold
            proxy_features.append({
                'feature': features_df.drop(columns=[sensitive_attr]).columns[i],
                'correlation': corr
            })
    
    return sorted(proxy_features, key=lambda x: x['correlation'], reverse=True)

# Usage
proxies = calculate_proxy_correlation(training_data, 'protected_attribute')
for proxy in proxies:
    print(f"Feature '{proxy['feature']}' correlates {proxy['correlation']:.2f} with protected attribute")

Once you've identified proxies, you have choices. Sometimes you remove them. Sometimes you keep them because they're genuinely predictive of the business outcome independent of the protected attribute. Sometimes you keep the feature but add fairness constraints during training. The important part is making that decision intentionally and documenting it.

Document intended use and limits

Every model has boundaries. A churn model trained on one market doesn't work for another. A fraud detector trained during normal times fails during crises. A content moderation model trained for one language community won't perform on another. Yet teams often deploy models far beyond their intended scope.

We shipped a customer propensity model for a SaaS company that performed beautifully on the enterprise segment where we trained it. The sales team wanted to use it for SMBs. We pushed back, documented the training scope, and created a model card that explicitly stated: "This model was trained on enterprise accounts with ARR > $100k. Do not use for SMB predictions without retraining."

A model card should include:

• Intended use: "Predict customer churn for enterprise SaaS customers in North America"
• Performance metrics: Broken down by segment, geography, time period
• Training data characteristics: Date range, demographics, edge cases included/excluded
• Known limitations: "Model does not account for competitive switching" or "performance degrades 30% on new market entrants"
• Harmful use cases: "Do not use for hiring decisions" or "Do not use to deny service"

This isn't just documentation for compliance. It's institutional memory that prevents the next team from making the same mistakes you did. A model card lives alongside your model in version control. When it changes, that's a deployment decision worth reviewing.

Review decisions with real impact

The higher the stakes, the more rigor you need. A recommendation algorithm that occasionally surfaces suboptimal content is low-stakes. A lending model that denies mortgages, a hiring model that filters resumes, or a risk assessment model that flags suspects for investigation—these are high-stakes.

For high-stakes decisions, we implement what we call "human review gates." The model makes a recommendation, but humans review decisions at the boundary—borderline approvals, any rejections, decisions that deviate from historical norms. This catches fairness problems before they affect users.

In a credit decisioning system we built, the model recommended "decline" for a small segment of applicants with strong income but recent address changes. A human reviewer would have immediately spotted the pattern: the model was biased against people going through life transitions. We added features to explicitly account for this, rebalanced our training data, and added a review gate for exactly this scenario.

Document these gates in your model serving pipeline:

def score_application(features):
    """Score an application with fairness-aware review gates."""
    
    score = model.predict(features)
    review_required = False
    review_reason = None
    
    # High-stakes decisions always get reviewed
    if score < 0.3 or score > 0.7:
        review_required = True
        review_reason = "Extreme prediction requires human validation"
    
    # High uncertainty
    if abs(score - 0.5) < 0.15:
        review_required = True
        review_reason = "Model uncertain, human judgment recommended"
    
    # Patterns that correlate with fairness concerns
    if features['recent_address_changes'] > 3:
        review_required = True
        review_reason = "Multiple address changes—potential life transition, manual review recommended"
    
    return {
        'score': score,
        'review_required': review_required,
        'review_reason': review_reason
    }

These review gates slow down decisions slightly, but they prevent systemic harm. More importantly, they give you signal about where your model is failing. If reviewers consistently override the model for a particular segment, that's a training data or labeling problem worth investigating.

Common mistakes

Assuming balanced accuracy means fair performance: A model can achieve 90% accuracy overall while having wildly different performance for minority groups. Always slice your metrics.

Only checking model performance, not real-world impact: A recommendation model might have fair performance metrics but still create filter bubbles. Fairness isn't just about model accuracy—it's about outcomes.

Treating fairness as a training problem only: The fairness of a system depends on data collection, labeling, model training, deployment boundaries, and human review. You can't solve it with a fairness library alone.

Documenting fairness once and moving on: Model behavior changes. User demographics change. Business requirements change. Fairness is something you monitor continuously, not a checkbox you mark on launch day.

Assuming fairness means treating everyone the same: Sometimes fairness requires acknowledging differences and building different models or adjusting thresholds for different groups. Formal equality isn't the same as real equity.

Key takeaways

• Performance metrics hide bias. Always measure accuracy, precision, recall, and other key metrics separately for each demographic group or user segment.
• Removing sensitive columns doesn't remove bias. Identify and deliberately decide how to handle proxy features that correlate with protected attributes.
• Document what your model is trained for and what it isn't. A model card saves future teams from applying your model to use cases where it fails.
• High-stakes decisions benefit from human review gates. Flag borderline predictions and anomalies for human validation before they affect users.
• Fairness isn't a one-time effort. Monitor performance continuously, review decisions periodically, and update your approach as you learn where your model causes real-world harm.