Towards A Rigorous Science of Interpretable Machine Learning
Finale Doshi-Velez, Been Kim
stat.ML, cs.AI, cs.LG
2017-02-28
A 2017 position paper defines interpretability as making a model understandable to a human, traces the need to incomplete specs, and lays out a three-level evaluation taxonomy.
By 2017, interpretability papers were easy to find and hard to compare. Google Scholar already returned more than 20,000 hits from the previous five years. Evaluation mostly split two ways. One camp dropped a method into an application, or a toy version of one, and treated usefulness as proof of interpretability; LIME-style work often sat here. The other camp declared a model class interpretable (sparse linear models, rule lists, gradient boosted trees) and then optimized inside that class. Both rest on face validity: a person looks at the output and it feels right.
That leaves questions the field could not score. Are all members of an "interpretable" class equally interpretable? How do you compare sparsity in features with sparsity in prototypes? Do all applications even want the same kind of explanation? The EU GDPR was also about to require a "right to explanation" by 2018 for algorithmic decisions that significantly affect users, while the measurement standard was still missing.
This is a position paper, not a new algorithm. Doshi-Velez and Kim want an operational definition and a map of evaluations that can actually be compared.
Interpretability is defined as the ability to explain or present in terms a human can understand. Psychology still does not agree on what counts as an explanation, so the paper does not close that debate. It asks for data-driven operational tests instead.
Interpretability is usually a tool, not the end goal. Fairness, privacy, reliability, robustness, causality, usability, and trust are the actual desiderata. Fairness and privacy already had formal criteria by 2017 and did not need to detour through explanations. For the rest, an explanation is a way to see whether those properties hold. An explanation that fails to match a causal structure is a warning, not a feature.
Many systems do not need this. Ad servers, postal-code sorters, and aircraft collision avoidance can skip explanations: either a bad output is cheap, or the problem has been validated in real deployments well enough that people already trust the system. The paper's claim is that explanation is needed when the problem specification is incomplete. Incompleteness is not uncertainty. A fused missile-location estimate can be uncertain and still be handled with quantified variance. Incompleteness produces unquantified bias, such as the unknown effect of injecting domain knowledge into model selection.
Five gaps keep showing up. Scientific understanding has no complete statement of what counts as knowledge. Safety-critical systems cannot enumerate every failure case or flag every bad output. Ethics often cannot encode fairness fully; protected classes that were written in still miss biases that appear only after training. Objectives get replaced by proxies, such as a clinic system that optimizes cholesterol without modeling adherence. Two fully specified goals, such as privacy versus accuracy, can still lack a known trade-off curve, so the cut has to be made case by case.
Evaluation is split into three rungs, more expensive and more task-specific as you go up.
Application-grounded tests put real humans on real tasks: doctors diagnosing, users correcting a segmentation. The baseline is whether human-written explanations help other humans do the same job. Human-grounded tests keep real humans but simplify the task, so lay participants can stand in for scarce experts. Three templates: forced choice between two explanations; forward simulation, where a person sees an explanation and an input and must predict the model's output; and counterfactual simulation, where the person is asked what to change to flip the prediction. The classic topic-model word-intrusion test is treated as forward simulation. Functionally-grounded tests skip humans and use a proxy such as sparsity or tree depth. That is appropriate only after the model class or regularizer has already been validated with people, or when the method is too immature (or too unethical) for a user study.
The three rungs have to feed each other. The proposed instrument is a matrix with real tasks on the rows, methods on the columns, and end-task performance in the cells, then matrix factorization to recover latent factors of interpretability, in the style of collaborative filtering. The paper also wants public repositories of explanation tasks that include a human-assessment loop, analogous to UCI, OpenML, and OpenAI Gym at the time.
Hypothesized task factors: global versus local; where the incompleteness sits and how severe it is; how much time the user has; how expert the user is, which changes their cognitive chunks. Hypothesized method factors: whether chunks are raw features, semantically meaningful derived features, or prototypes; how many chunks; how they compose (rules, hierarchies); monotonicity; how well people handle uncertainty and stochasticity.
The practical rule for researchers: match the claim to the rung. An application paper should run application-grounded or closely related human-grounded tests. A paper that only improves optimization inside an already-accepted interpretable class can stay functionally-grounded. Citations should tag the same four things: where the spec is incomplete, which evaluation rung was used, task factors, and method factors.
No new model, and no accuracy table against LIME or sparse linear baselines. The deliverable is the evaluation map plus the external constraints the authors could pin down in 2017.
| Item | Content |
| Paper type | Position paper, 13 pages |
| Evaluation rungs | Application-, human-, and functionally-grounded |
| When to skip explanation | Low-stakes errors, or a problem already validated in deployment |
| Root cause | Incomplete formalization, not quantified uncertainty |
| Regulatory marker | GDPR explanation duty aimed at 2018 |
| Literature volume | More than 20,000 related papers in five years on Google Scholar |
The three human-grounded experiments are templates, not newly measured scores. Functionally-grounded metrics are explicitly demoted: sparsity can compare models that share a representation, but feature sparsity and prototype sparsity do not share a scale, and neither is an axiom until humans have validated it. The baseline being criticized is evaluation practice itself: "it helped in an app" and "we optimized an interpretable class" are tagged as face validity only.
This taxonomy became the default coordinate system for later XAI evaluation talk. Plenty of later papers still live on the bottom rung: they report sparsity, attention alignment, or fidelity to a black box, and almost never put doctors, lawyers, or safety engineers on the real task. Mismatch between claim and evaluation rung is exactly what this paper was trying to block.
For practitioners the cut is operational. A product that has to pass compliance or a clinic needs application-grounded evidence; proxy metrics are not enough. A new explanation format can start with forward or counterfactual simulation on lay users, which is cheaper than recruiting experts. Tightening sparsity inside a class already shown to be interpretable can stay on functional proxies.
The other still-useful question is whether you need an explanation at all. Ad ranking and a validated collision-avoidance stack can skip it. Systems whose objective is a proxy, whose failure cases cannot be listed, or whose fairness definition will not fit in a loss, need explanations as a way to surface the gap, not as decoration.
This is a framework paper. It does not replace SHAP. It does stop a review from treating sparsity as interpretability.
The latent factors are hypotheses. The task-method matrix was never built. The repository is a proposal; no dataset or platform shipped with the paper.
There is no conversion between rungs. Success on an application does not yield a human-grounded score, and a human-grounded score does not yield a sparsity threshold. Without that mapping, functionally-grounded papers can still call themselves interpretable, and the critique has no numerical teeth.
The definition depends on "understandable to a human," and humans are not one user. A clinician and a hospital ethicist want different grain. The paper flags this and does not give a measurement protocol.
The 2017 external anchors have aged. Legal scholars later argued over whether GDPR actually mandates algorithmic explanations; this paper uses the regulation as urgency evidence and does not track that dispute. "20,000 papers" is a Google Scholar retrieval count with a wide query, not a census of the field.
As a position paper it never scores a concrete method with its own taxonomy. LIME appears only as an example of application-context evaluation, not as a method placed on the three rungs. The framework is not demonstrated on a worked example.