A1 · Supervised Machine Learning in Depth

Model Interpretability

Model interpretability methods explain what a model has learned. Global methods describe overall behaviour: permutation importance measures how much the score drops when a feature's values are shuffled, and partial dependence shows how the average prediction changes with a feature. Local methods explain single predictions: SHAP values split a prediction into additive contributions from each feature, based on Shapley values from game theory.

15 min Advanced Jump to the activity

What you need: The earlier topics of this module, and its prerequisites.

Definition

Why was this loan application rejected? Can the model explain itself?

Model interpretability methods explain what a model has learned. Global methods describe overall behaviour: permutation importance measures how much the score drops when a feature's values are shuffled, and partial dependence shows how the average prediction changes with a feature. Local methods explain single predictions: SHAP values split a prediction into additive contributions from each feature, based on Shapley values from game theory.

In simple words

Shuffle one column, say income, so it becomes meaningless. If accuracy collapses, the model relied on income. For one customer, SHAP says something like: the average approval score is 0.6; low income pulled it down by 0.2; a long credit history pushed it up by 0.1; final score 0.5.

Why it matters

Explanations are needed to debug models, catch leakage and bias, build trust with users, and meet legal duties to explain automated decisions in areas such as lending, hiring and healthcare.

Key terms

Global interpretability
Understanding how a model behaves overall: which features matter and how.
Local interpretability
Explaining one prediction, such as why a particular loan was refused.
Permutation importance
The drop in a validation metric when one feature's values are shuffled.
Partial dependence plot (PDP)
The average prediction as one feature varies, with others at their observed values.
SHAP values
Contributions of each feature to a prediction, based on Shapley values, summing to the prediction minus the average.
Surrogate model
A simple model, such as a shallow tree, trained to imitate a complex one for explanation.

Explained in detail

Why interpret a model?

Interpretation serves several goals: debugging, by revealing that a model relies on a leaked or spurious feature; trust, by showing domain experts that predictions follow sensible patterns; fairness, by checking whether sensitive attributes or their proxies drive decisions; and accountability, by giving people affected by a decision understandable reasons, as many regulations expect. Simple models such as linear regression and shallow trees are interpretable directly. For random forests, gradient boosting and neural networks we use model-agnostic tools that treat the model as a black box and probe it with data.

The main tools

ToolAnswersCaveat
Permutation importanceWhich features does the model rely on overall?Correlated features share importance and may each look unimportant
Partial dependence and ICE plotsHow does the prediction change as one feature changes?Can show impossible combinations when features are correlated
SHAPHow much did each feature push this prediction up or down?Explains the model, not the real-world cause
Surrogate treeWhat simple rules approximate the model?Check how faithfully it mimics the model
CounterfactualsWhat would need to change to get a different outcome?Suggested changes must be feasible and actionable

SHAP, introduced by Scott Lundberg and Su-In Lee in 2017, rests on Shapley values from cooperative game theory, which divide a team's payout fairly among players according to their average marginal contribution over all orders of joining. For a loan model, a SHAP explanation might say: the average predicted default risk is 12 per cent; this applicant's risk is 31 per cent; two missed payments added 14 points, a short employment history added 7, and high income subtracted 2. TreeSHAP computes exact values quickly for tree ensembles. Summary plots of SHAP values across many rows give a global picture of importance and direction.

Interpretations describe what the model does, not what causes outcomes in the world, and they can be unstable when features are correlated. Use them on validation data, compare several methods, and discuss surprising findings with domain experts before acting on them.

How it works

MethodScopeAnswers
Coefficients (linear models)GlobalDirection and size of each effect
Permutation importanceGlobalWhich features the model relies on
Partial dependenceGlobalHow predictions change with one feature
SHAPLocal and globalWhy this prediction, feature by feature

The maths

Permutation importance of feature j
The score on intact data
The score after the k-th shuffle of feature j
SHAP: additive explanation of one prediction

In code

Permutation importance for a random forest on held-out data, and a partial dependence curve for the top feature. The shap library adds per-prediction explanations on your own machine.

interpretability.py
import numpy as np
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import partial_dependence, permutation_importance
from sklearn.model_selection import train_test_split

data = load_breast_cancer()
X_tr, X_te, y_tr, y_te = train_test_split(data.data, data.target, test_size=0.3, stratify=data.target, random_state=0)
model = RandomForestClassifier(n_estimators=200, random_state=0).fit(X_tr, y_tr)

result = permutation_importance(model, X_te, y_te, n_repeats=10, random_state=0)
order = np.argsort(result.importances_mean)[::-1][:5]
print("Top features by permutation importance (accuracy drop):")
for j in order:
    print(f"  {data.feature_names[j]:<22} {result.importances_mean[j]:.3f} +/- {result.importances_std[j]:.3f}")

top = order[0]
pd_result = partial_dependence(model, X_tr, [top], grid_resolution=5)
grid = pd_result["grid_values"][0]
for value, avg in zip(grid, pd_result["average"][0]):
    print(f"  {data.feature_names[top]} = {value:7.2f} -> average P(benign) {avg:.2f}")

Output

Top features by permutation importance (accuracy drop):
  worst radius           0.021 +/- 0.005
  worst area             0.019 +/- 0.007
  worst texture          0.016 +/- 0.004
  worst perimeter        0.012 +/- 0.007
  worst concave points   0.012 +/- 0.008
  worst radius =   10.47 -> average P(benign) 0.67
  worst radius =   14.23 -> average P(benign) 0.67
  worst radius =   18.00 -> average P(benign) 0.51
  worst radius =   21.77 -> average P(benign) 0.50
  worst radius =   25.54 -> average P(benign) 0.50

Try it yourself

Read the explanation

Match each question to the method that answers it.

  • Which features does the model rely on overall?
  • Why was this one application rejected?
  • How does the predicted risk change as age increases?
  • In a linear model, what does a one-unit increase do?

Common mistakes

  • Computing permutation importance on training data, which rewards memorised features.
  • Reading importances of correlated features separately. Shuffling one leaves its twin.
  • Treating explanations as causal proof.

Check yourself

Question 1

1. Permutation importance measures…

Question 2

2. SHAP values for one prediction add up to…

Question 3

3. A feature has high importance. Does it cause the outcome?

Summary

  • Global methods: coefficients, permutation importance, partial dependence.
  • Local method: SHAP splits one prediction into feature contributions.
  • Compute on held-out data, beware correlation, and don't read causation.

Finished reading?

Sources: scikit-learn: Permutation importance · Lundberg and Lee, A Unified Approach to Interpreting Model Predictions (2017) · Molnar, Interpretable Machine Learning