I10 · Responsible AI in Practice

Explainable AI

Explainable AI (XAI) is the set of methods for making a model's behaviour understandable to people. Some models are interpretable by design, such as short decision trees, linear models and rule lists. For complex "black-box" models, post-hoc methods explain behaviour after training: global methods such as feature importance and partial dependence describe the model overall, while local methods such as LIME (2016) and SHAP (2017) explain an individual prediction by estimating how much each feature pushed it up or down. Counterfactual explanations describe the smallest change that would alter the outcome, such as "with an income ₹5,000 higher, the loan would be approved".

12 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

"Your loan application has been rejected." Why? If the bank cannot answer, you cannot correct a mistake, improve your chances or challenge an unfair decision. Explainable AI tries to answer the question "why did the model decide that?"

Explainable AI (XAI) is the set of methods for making a model's behaviour understandable to people. Some models are interpretable by design, such as short decision trees, linear models and rule lists. For complex "black-box" models, post-hoc methods explain behaviour after training: global methods such as feature importance and partial dependence describe the model overall, while local methods such as LIME (2016) and SHAP (2017) explain an individual prediction by estimating how much each feature pushed it up or down. Counterfactual explanations describe the smallest change that would alter the outcome, such as "with an income ₹5,000 higher, the loan would be approved".

In simple words

A good teacher does not just give marks; they explain what you got right, what lost marks and what to do next time. Explainable AI tries to make models behave like that teacher.

Overview

  • Explainable AI (XAI) means methods that help people understand why a model made a prediction.
  • Complex models such as deep networks and large ensembles are often called "black boxes".
  • Explanations can be global (how the model works overall) or local (why this one prediction).
  • Popular methods include feature importance, SHAP, LIME, saliency maps and counterfactuals.
  • Explanations help debug models, build trust, meet laws and let people challenge decisions.
  • Explanations can be misleading, so they should be tested and presented carefully.

Why it matters

Explanations help people trust the right models and distrust the wrong ones, let affected people contest decisions, help developers find bugs and shortcuts, and are increasingly expected by regulators for high-stakes decisions.

Key terms

Interpretable model
A model simple enough that people can follow how it reaches decisions.
Global explanation
A description of which features matter for the model overall.
Local explanation
Reasons for one particular prediction.
Permutation importance
How much performance drops when a feature's values are shuffled.
SHAP values
Per-feature contributions to one prediction, based on game theory, that add up to the difference from the average prediction.
Counterfactual explanation
The smallest change in inputs that would change the decision.

Types

Explainability methods come in several types. They differ in scope and in how they work.

1. Intrinsically interpretable models

Models simple enough to understand directly, such as linear models, small decision trees and rule lists.

  • How it works: Coefficients or rules show exactly how each feature affects the output.
  • Example: A credit score where each extra late payment lowers the score by a known amount.
  • When to use it: Use them when decisions must be fully explainable and accuracy is close to complex models.

2. Global feature importance

Ranking which features matter most to the model overall.

  • How it works: Permutation importance shuffles one feature at a time and measures how much the score drops.
  • Example: Finding that attendance and past marks drive a dropout-risk model most.
  • When to use it: Use it to understand and debug a model and to spot suspicious features.

3. Local explanations: SHAP and LIME

Explaining one prediction by showing how much each feature pushed it up or down.

  • How it works: SHAP (Lundberg and Lee, 2017) uses Shapley values from game theory; LIME (Ribeiro et al., 2016) fits a simple model around the prediction.
  • Example: A loan explanation: income +0.12, existing debt −0.20, credit history +0.05.
  • When to use it: Use them to explain individual decisions to users, auditors and regulators.

4. Saliency and attention maps

Highlighting which pixels or words most influenced a deep model's output.

  • How it works: Gradient-based methods such as Grad-CAM compute how sensitive the output is to each part of the input.
  • Example: A heat map showing an X-ray model focused on a ruler mark instead of the lungs, revealing a shortcut.
  • When to use it: Use them to check image and text models are looking at the right evidence.

5. Counterfactual and example-based explanations

Explaining with "what would need to change" or with similar past cases.

  • How it works: A counterfactual finds the smallest change to inputs that flips the decision; example-based methods show the most similar training cases.
  • Example: "Your loan would be approved if your monthly income were ₹5,000 higher."
  • When to use it: Use them for people affected by decisions, because they suggest actionable steps.

Explained in detail

Interpretable by design or explained afterwards

Cynthia Rudin of Duke University has argued that for high-stakes decisions we should prefer models that are interpretable by design, because post-hoc explanations of black boxes can be incomplete or misleading. For many tabular problems, such as credit scoring with a dozen features, a well-built logistic regression or a small decision tree performs nearly as well as a complex model, and every decision can be traced. For images, speech and text, deep networks are usually far more accurate, so post-hoc methods are needed. Saliency maps highlight which pixels most influenced an image classification; attention and token attributions highlight words that influenced a text model. These can reveal shortcuts: the 2016 LIME paper by Marco Tulio Ribeiro and colleagues showed a classifier that distinguished wolves from huskies largely by detecting snow in the background.

Choosing an explanation method

MethodAnswersGood for
Coefficients or tree rulesHow the whole model worksSimple, interpretable models
Permutation importanceWhich features matter most overallAny model; quick global check
SHAP or LIMEWhy this particular predictionExplaining individual decisions to users or auditors
CounterfactualsWhat would need to change for a different outcomeGiving people actionable advice
Saliency mapsWhich image regions matteredSpotting shortcuts in vision models

Good explanations for people

  • Match the audience: a farmer needs two or three plain reasons; an auditor may need full technical detail.
  • Make explanations actionable, such as which missing document or which payment history would change the result.
  • Be honest about uncertainty and limits; an explanation of a model is not proof that the model is right.
  • Check explanations for leaks of private data or of details that would let people game the system.

Explanations also matter legally. Europe's General Data Protection Regulation gives people rights around solely automated decisions with significant effects, including meaningful information about the logic involved, and the EU AI Act adds transparency duties for high-risk systems. In India, the principle of natural justice, that a person should know the reasons for a decision against them and have a chance to respond, applies strongly to government decisions, so public agencies using AI need to be able to explain outcomes. Even where the law is silent, explanations build trust and help organisations catch mistakes before they spread.

How it works

  1. Prefer an interpretable model when it performs nearly as well.
  2. Use permutation importance to see which features drive the model overall.
  3. Use SHAP, LIME or counterfactuals to explain individual decisions.
  4. Look for suspicious features or shortcuts, such as proxies for sensitive attributes.
  5. Turn explanations into plain, actionable messages for the people affected.

In code

Income and repayment history matter most. Shuffling pin_zone changes test accuracy by about zero (a slightly negative value is just noise), confirming that the model does not need it. Its small logistic coefficient is also noise from a limited sample; with real data you would check whether such a feature is acting as a proxy.

explain_model.py
import numpy as np
from sklearn.ensemble import RandomForestClassifier
from sklearn.inspection import permutation_importance
from sklearn.linear_model import LogisticRegression

rng = np.random.default_rng(0)
n = 600
income = rng.normal(40, 12, n)             # thousand rupees per month
repaid_before = rng.integers(0, 2, n)
pin_zone = rng.integers(0, 2, n)           # a feature that should not matter
noise = rng.normal(0, 6, n)
approve = ((income + 15 * repaid_before + noise) > 50).astype(int)
X = np.column_stack([income, repaid_before, pin_zone])
names = ["income", "repaid_before", "pin_zone"]

X_tr, X_te, y_tr, y_te = X[:400], X[400:], approve[:400], approve[400:]
forest = RandomForestClassifier(n_estimators=100, random_state=0).fit(X_tr, y_tr)
imp = permutation_importance(forest, X_te, y_te, n_repeats=10, random_state=0)   # on held-out data
print("Permutation importance on test data (drop in accuracy when shuffled):")
for name, value in sorted(zip(names, imp.importances_mean), key=lambda t: -t[1]):
    print(f"   {name:<14} {value:.3f}")

# An interpretable model: coefficients are the explanation
logit = LogisticRegression(max_iter=1000).fit(X_tr, y_tr)
print("Logistic regression coefficients:", {n: round(float(c), 2) for n, c in zip(names, logit.coef_[0])})

# A counterfactual for one rejected applicant: how much more income would flip the decision?
applicant = np.array([[38.0, 0, 1]])
print("Decision now:", "approve" if logit.predict(applicant)[0] else "reject")
for extra in range(0, 31, 2):
    if logit.predict(applicant + [[extra, 0, 0]])[0] == 1:
        print(f"Counterfactual: with Rs {extra},000 more monthly income the model would approve.")
        break

Output

Permutation importance on test data (drop in accuracy when shuffled):
   income         0.163
   repaid_before  0.105
   pin_zone       -0.035
Logistic regression coefficients: {'income': 0.24, 'repaid_before': 2.99, 'pin_zone': 0.46}
Decision now: reject
Counterfactual: with Rs 10,000 more monthly income the model would approve.

Try it yourself

The wolf detector that saw snow

Researchers trained an image classifier to tell wolves from husky dogs.

Match the question to the method

Which XAI method answers each question?

  • Which features matter most overall?
  • Why was this applicant rejected?
  • What change would get me approved?
  • Which pixels made it say "pneumonia"?

Common mistakes

  • Using a black box when an interpretable model would do.
  • Treating a plausible-looking explanation as proof the model is correct.
  • Giving technical explanations to people who need plain, actionable reasons.

Check yourself

Question 1

1. What does a counterfactual explanation describe?

Question 2

2. Permutation importance measures…

Question 3

3. The wolf-versus-husky example showed that…

Summary

  • Explainable AI makes models understandable through interpretable designs and post-hoc methods.
  • Use global methods for overall behaviour, local methods and counterfactuals for individual decisions.
  • Explanations reveal shortcuts and support appeals, but must be honest and suited to their audience.

Finished reading?

Sources: Christoph Molnar, Interpretable Machine Learning · Ribeiro, Singh and Guestrin (2016), "Why Should I Trust You?" Explaining the Predictions of Any Classifier