interpretability

Interpretability asks why a model produced a specific output, and the field's central, uncomfortable tension is between an explanation that looks convincing and an explanation that's actually true — plausibility and fidelity are different properties, and most popular explanation methods are much better validated on the first than the second. An explanation can be visually compelling, internally consistent, and still fail to describe what the model actually computed, which is why interpretability research spends as much effort testing whether explanations are faithful as it does producing them.

LIME and SHAP approach the same local-explanation problem from different mathematical starting points. Ribeiro and colleagues' LIME (2016) explains one specific prediction by perturbing the input slightly, observing how the model's output changes, and fitting a simple, interpretable model — usually a linear one — to approximate the complex model's behavior in just that local neighborhood around the prediction being explained. Lundberg and Lee's SHAP (2017) instead grounds feature attribution in cooperative game theory, computing each feature's Shapley value — its average marginal contribution across every possible combination of features it could appear with or without — which gives SHAP values a mathematical guarantee LIME's local linear fit doesn't have: attributions that provably sum to the exact difference between the prediction and a baseline. The cost of that guarantee is computational — exact Shapley values require evaluating an exponential number of feature subsets, which is why practical SHAP implementations use approximations rather than the exact computation for anything but small feature sets.

Why did the model produce this output?
the question interpretability tries to answer
Is the stakes high enough to need a guarantee, not just a plausible story?
High-stakes decision (medical, legal, safety-critical)
Rudin (2019): use an inherently interpretable model, not a post-hoc explanation of a black box
Lower-stakes, model already deployed
a post-hoc explanation of the existing black box is acceptable
Post-hoc: local or global?
Local — explain one prediction
LIME (local surrogate) or SHAP (Shapley values)
Global — explain the model's overall behavior
feature importance aggregated across many predictions
Validate: does the explanation reflect the real computation?
plausibility is not the same thing as fidelity

Attention weights seemed like a free, built-in explanation, until researchers tested whether they actually were one. Transformer and earlier attention-based models expose attention weights as a natural-looking number for how much the model "focused on" each input token, and for years that number was casually treated as an explanation of the model's reasoning. Jain and Wallace (2019) tested this directly and found that attention weights often don't correlate with the features that actually matter for the prediction — you could substitute a very different attention pattern and get the same output, or find that features with low attention weight were nonetheless the ones driving the result when tested by direct intervention. This was an important correction for the field specifically because attention is so visually intuitive and so cheap to produce — a heatmap over input tokens looks like an explanation whether or not it functions as one, and the gap between looking like an explanation and being one is exactly the plausibility-versus-fidelity problem the whole field has to keep checking for.

A plausible explanation and a faithful one are not the same claim Plausible "this explanation looks reasonable to a human reviewer" an attention heatmap, a SHAP bar chart, a LIME local fit — all visually convincing judged by how it looks, not verified can be wrong and still look right Faithful "this explanation accurately reflects the model's actual computation" tested by intervention: change the feature the explanation points to, check whether the output actually changes verified, not just visually judged Jain and Wallace (2019) showed attention weights often fail this test even when they look plausible.

Rudin (2019) made the strongest version of the argument against post-hoc explanation entirely: for high-stakes decisions, build an interpretable model instead of explaining an uninterpretable one. Her central claim is that post-hoc explanations are themselves separate models approximating the real one, and an approximation of a black box can be arbitrarily unfaithful to it while still looking reasonable — so for decisions with real consequences, like parole or medical diagnosis, the responsible choice is an inherently interpretable model (a sparse linear model, a small decision tree, a rule list) where the explanation and the actual computation are the same thing, not a complex black box paired with a separate, possibly-wrong explanation bolted on afterward. This is a genuinely contested position in the field — inherently interpretable models often sacrifice some accuracy compared to a large black-box model — but it reframes interpretability from "how do we explain this model" to "did this decision need an uninterpretable model in the first place."

Read interpretability as an evidentiary standard, not a visualization feature: an explanation method is a claim about what a model is doing, and that claim needs the same scrutiny any other empirical claim would — tested by intervention, not accepted because the output looked reasonable — which is why the field's real progress has come less from producing prettier explanations and more from developing ways to catch the ones that were never true in the first place.

Go deeper with CFSGPT

Get AI-powered deep-dives, save terms, and run advanced simulations — free account.

Create Free Account