Explainable AI with SHAP: a AI & ML guide for developers

Explainable AI with SHAP: a AI & ML guide for developers. Understand why a model made a prediction.

Why explain — Debugging, trust and regulation

In production machine learning, a high accuracy score tells you something works, but it doesn't tell you why. When a model denies a loan, recommends the wrong product, or flags a transaction as fraud, stakeholders need answers. Explainable AI with SHAP provides a principled way to understand model predictions at scale.

Consider a credit risk model that rejects an applicant. The business team needs to explain the decision to the customer. The compliance officer needs documentation for regulatory audit. The data scientist needs to verify the model isn't picking up on proxies for protected attributes. A single accuracy metric answers none of these questions.

SHAP (SHapley Additive exPlanations) gives us a mathematically rigorous framework grounded in game theory. Every prediction is decomposed into feature contributions that sum to the model output. This isn't just useful for debugging—it's increasingly mandatory. Regulations like GDPR's right to explanation and fair lending laws require documentation of decision factors.

Without explainability, you're flying blind. With it, you gain visibility into model behavior, catch data leakage, identify bias, and build user trust. For teams shipping models to production, this is foundational.

SHAP values — Contribution of each feature to a prediction

SHAP values measure how much each feature contributes to pushing a prediction away from the model's average output. The base value is the expected prediction when you know nothing. Each feature then adds or subtracts from that base value. The sum of all contributions equals the final prediction.

Mathematically, SHAP values come from cooperative game theory. Imagine features as players in a coalition game. A SHAP value for a feature represents its average marginal contribution when added to all possible subsets of other features. This ensures fair attribution—a feature only gets credit for what it actually contributes, not for correlations with other features.

Let's walk through a concrete example. Say we're predicting house prices with a random forest model:

import shap
import xgboost as xgb
from sklearn.datasets import fetch_california_housing
import pandas as pd

# Load data
housing = fetch_california_housing()
X = pd.DataFrame(housing.data, columns=housing.feature_names)
y = housing.target

# Train model
model = xgb.XGBRegressor().fit(X, y)

# Create SHAP explainer
explainer = shap.TreeExplainer(model)
shap_values = explainer.shap_values(X)

# Examine one prediction
instance_idx = 0
print(f"Base value (expected model output): {explainer.expected_value}")
print(f"Actual prediction: {model.predict(X[[instance_idx]])[0]}")
print(f"Feature contributions: {dict(zip(X.columns, shap_values[instance_idx]))}")

The output shows the base value (average house price), then how each feature pushed the prediction up or down. A feature like high latitude might add $50k, while high crime rate might subtract $30k. The sum of all contributions plus the base value gives you the final prediction.

Different model types need different explainers. XGBoost and LightGBM models use TreeExplainer for fast computation. Linear models use LinearExplainer. Neural networks and general models use KernelExplainer, which is slower but model-agnostic.

Global view — Summary plots show the most important features overall

Local explanations answer "why this prediction?" Global explanations answer "which features matter most across all predictions?" This view helps you understand overall model behavior and spot issues before they reach customers.

The summary plot shows feature importance ranked by the magnitude of SHAP values. More precisely, it sums the absolute contribution of each feature across all instances. Features that consistently push predictions in either direction rank higher.

import matplotlib.pyplot as plt

# Summary plot (bar chart)
shap.summary_plot(shap_values, X, plot_type="bar", show=False)
plt.title("Average impact of features on model output")
plt.tight_layout()
plt.savefig("shap_summary_bar.png", dpi=150, bbox_inches="tight")

# Summary plot (beeswarm - more detailed)
shap.summary_plot(shap_values, X, show=False)
plt.title("SHAP values per feature")
plt.tight_layout()
plt.savefig("shap_summary_beeswarm.png", dpi=150, bbox_inches="tight")

The beeswarm plot adds color intensity. Each dot is one instance. The horizontal position shows the SHAP value (contribution magnitude), and the color shows the feature value. Red means high feature value, blue means low. Patterns emerge: if latitude shows all red dots on the left and blue on the right, you know high latitude consistently decreases predictions, and that effect is strong.

This global view catches model issues. If a feature you expected to be important barely registers, something's wrong—maybe the data pipeline broke it. If an unexpected proxy variable dominates, you've found a fairness risk before production.

Local view — Explain individual decisions to users or support teams

Global insights set strategy. Local explanations handle individual customer interactions. When a support agent fields a complaint—"Why was I rejected?"—you need a clear, customer-friendly explanation of that specific decision.

The force plot visualizes one prediction by showing each feature contribution as a colored block. Push right from base value, you get the final prediction. This is intuitive for non-technical audiences.

import shap

# Force plot for one instance
instance_idx = 5
shap.force_plot(
    explainer.expected_value,
    shap_values[instance_idx],
    X.iloc[instance_idx],
    show=False
).savefig("shap_force_plot.html")

# For programmatic access, convert to text explanation
print(f"Prediction: {model.predict(X[[instance_idx]])[0]:.2f}")
print(f"Base value: {explainer.expected_value:.2f}")
print("\nFactors pushing prediction UP:")
for feature, shap_val in sorted(zip(X.columns, shap_values[instance_idx]), 
                                 key=lambda x: x[1], reverse=True)[:5]:
    if shap_val > 0:
        print(f"  {feature}: +{shap_val:.2f} (value: {X.iloc[instance_idx][feature]:.2f})")

print("\nFactors pushing prediction DOWN:")
for feature, shap_val in sorted(zip(X.columns, shap_values[instance_idx]), 
                                 key=lambda x: x[1])[:5]:
    if shap_val < 0:
        print(f"  {feature}: {shap_val:.2f} (value: {X.iloc[instance_idx][feature]:.2f})")

You can now generate customer-facing statements: "Your application score was 0.65. Your credit history added +0.12, employment stability added +0.08, but recent account inquiries subtracted -0.15." It's factual, specific, and auditable.

For a lending team, you might build a dashboard where loan officers see the top three reasons for accept/reject recommendations. This accelerates review, improves consistency, and creates a paper trail for compliance.

Local explanations also integrate into support systems. Pair them with customer data to identify if a rejection was fair or whether a data entry error skewed things. A customer might realize they entered an employment date wrong and ask for reconsideration with corrected info.

Common mistakes

• Using KernelExplainer on large datasets without sampling. KernelExplainer creates many model calls to estimate contributions. On 1M rows, it's prohibitively slow. Sample 5-10k rows or use TreeExplainer for tree-based models instead.
• Interpreting SHAP values as feature importance in causal terms. SHAP shows correlation-based contributions. High SHAP doesn't mean the feature causes the outcome. If income and zip code are correlated, one might steal credit from the other depending on model behavior.
• Sharing raw SHAP values with non-technical stakeholders. "Latitude contributed -0.237" means nothing to a customer. Translate into ranges and business language: "Living more than 50 miles from the city center reduced the score."
• Computing SHAP values once and assuming they're stable. Retrain your model, or retrain on new data? Recompute SHAP values. Feature contributions shift with model weights.
• Ignoring class imbalance effects in classification. If your fraud model has 1% positive class, SHAP values reflect that skew. A large positive SHAP might still predict "not fraud" if the base value (prior) is heavily skewed toward legitimate.

Key takeaways

• Explainable AI with SHAP is no longer optional—regulatory and business requirements demand it. Use SHAP to document decisions for compliance and build user trust.
• SHAP values decompose predictions into fair feature contributions grounded in game theory. Every feature gets credit only for what it genuinely contributes, not for correlated features.
• Global SHAP summaries show which features dominate model behavior overall. Use them to catch data quality issues, unexpected proxies, and fairness risks before production.
• Local SHAP explanations enable individual decision auditing and customer communication. Convert SHAP values into business-friendly language for support teams and compliance reviews.
• Pick the right explainer type: TreeExplainer for tree models (fast), LinearExplainer for linear models, KernelExplainer for anything else (slower, requires sampling on large datasets).
• SHAP values show correlation-based contributions, not causation. Interpret carefully and refresh them after model retraining.