Definition
Why was this loan application rejected? Can the model explain itself?
Model interpretability methods explain what a model has learned. Global methods describe overall behaviour: permutation importance measures how much the score drops when a feature's values are shuffled, and partial dependence shows how the average prediction changes with a feature. Local methods explain single predictions: SHAP values split a prediction into additive contributions from each feature, based on Shapley values from game theory.
In simple words
Shuffle one column, say income, so it becomes meaningless. If accuracy collapses, the model relied on income. For one customer, SHAP says something like: the average approval score is 0.6; low income pulled it down by 0.2; a long credit history pushed it up by 0.1; final score 0.5.
Why it matters
Explanations are needed to debug models, catch leakage and bias, build trust with users, and meet legal duties to explain automated decisions in areas such as lending, hiring and healthcare.
Key terms
- Global interpretability
- Understanding how a model behaves overall: which features matter and how.
- Local interpretability
- Explaining one prediction, such as why a particular loan was refused.
- Permutation importance
- The drop in a validation metric when one feature's values are shuffled.
- Partial dependence plot (PDP)
- The average prediction as one feature varies, with others at their observed values.
- SHAP values
- Contributions of each feature to a prediction, based on Shapley values, summing to the prediction minus the average.
- Surrogate model
- A simple model, such as a shallow tree, trained to imitate a complex one for explanation.
Explained in detail
Why interpret a model?
Interpretation serves several goals: debugging, by revealing that a model relies on a leaked or spurious feature; trust, by showing domain experts that predictions follow sensible patterns; fairness, by checking whether sensitive attributes or their proxies drive decisions; and accountability, by giving people affected by a decision understandable reasons, as many regulations expect. Simple models such as linear regression and shallow trees are interpretable directly. For random forests, gradient boosting and neural networks we use model-agnostic tools that treat the model as a black box and probe it with data.
The main tools
| Tool | Answers | Caveat |
|---|---|---|
| Permutation importance | Which features does the model rely on overall? | Correlated features share importance and may each look unimportant |
| Partial dependence and ICE plots | How does the prediction change as one feature changes? | Can show impossible combinations when features are correlated |
| SHAP | How much did each feature push this prediction up or down? | Explains the model, not the real-world cause |
| Surrogate tree | What simple rules approximate the model? | Check how faithfully it mimics the model |
| Counterfactuals | What would need to change to get a different outcome? | Suggested changes must be feasible and actionable |
SHAP, introduced by Scott Lundberg and Su-In Lee in 2017, rests on Shapley values from cooperative game theory, which divide a team's payout fairly among players according to their average marginal contribution over all orders of joining. For a loan model, a SHAP explanation might say: the average predicted default risk is 12 per cent; this applicant's risk is 31 per cent; two missed payments added 14 points, a short employment history added 7, and high income subtracted 2. TreeSHAP computes exact values quickly for tree ensembles. Summary plots of SHAP values across many rows give a global picture of importance and direction.
Interpretations describe what the model does, not what causes outcomes in the world, and they can be unstable when features are correlated. Use them on validation data, compare several methods, and discuss surprising findings with domain experts before acting on them.
How it works
| Method | Scope | Answers |
|---|---|---|
| Coefficients (linear models) | Global | Direction and size of each effect |
| Permutation importance | Global | Which features the model relies on |
| Partial dependence | Global | How predictions change with one feature |
| SHAP | Local and global | Why this prediction, feature by feature |
The maths
- The score on intact data
- The score after the k-th shuffle of feature j
In code
Permutation importance for a random forest on held-out data, and a partial dependence curve for the top feature. The shap library adds per-prediction explanations on your own machine.
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import partial_dependence, permutation_importance
from sklearn.model_selection import train_test_split
data = load_breast_cancer()
X_tr, X_te, y_tr, y_te = train_test_split(data.data, data.target, test_size=0.3, stratify=data.target, random_state=0)
model = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_tr, y_tr)
result = permutation_importance(model, X_te, y_te, n_repeats=10, random_state=0)
order = np.argsort(result.importances_mean)[::-1][:5]
print("Top features by permutation importance (accuracy drop):")
for j in order:
print(f" {data.feature_names[j]:<22} {result.importances_mean[j]:.3f} +/- {result.importances_std[j]:.3f}")
top = order[0]
pd_result = partial_dependence(model, X_tr, [top], grid_resolution=5)
grid = pd_result["grid_values"][0]
for value, avg in zip(grid, pd_result["average"][0]):
print(f" {data.feature_names[top]} = {value:7.2f} -> average P(benign) {avg:.2f}")Output
Top features by permutation importance (accuracy drop): worst radius 0.021 +/- 0.005 worst area 0.019 +/- 0.007 worst texture 0.016 +/- 0.004 worst perimeter 0.012 +/- 0.007 worst concave points 0.012 +/- 0.008 worst radius = 10.47 -> average P(benign) 0.67 worst radius = 14.23 -> average P(benign) 0.67 worst radius = 18.00 -> average P(benign) 0.51 worst radius = 21.77 -> average P(benign) 0.50 worst radius = 25.54 -> average P(benign) 0.50
Try it yourself
Read the explanation
Match each question to the method that answers it.
- Which features does the model rely on overall?
- Why was this one application rejected?
- How does the predicted risk change as age increases?
- In a linear model, what does a one-unit increase do?
Common mistakes
- Computing permutation importance on training data, which rewards memorised features.
- Reading importances of correlated features separately. Shuffling one leaves its twin.
- Treating explanations as causal proof.
Check yourself
Summary
- Global methods: coefficients, permutation importance, partial dependence.
- Local method: SHAP splits one prediction into feature contributions.
- Compute on held-out data, beware correlation, and don't read causation.
Finished reading?
Sources: scikit-learn: Permutation importance · Lundberg and Lee, A Unified Approach to Interpreting Model Predictions (2017) · Molnar, Interpretable Machine Learning