Pando: Do Interpretability Methods Work When Models Won't Explain Themselves?
Ziqian Zhong, Aashiq Muhamed, Mona T. Diab, Virginia Smith, Aditi Raghunathan
cs.LG, cs.AI
2026-04-13
CMU plants hidden trees in 720 Gemma-2-2B models. When they stay silent or lie, RelP beats a 10-sample black-box by 3-5 points; SAE and circuit tracing do not.
Mechanistic interpretability is sold as a way to audit models whose own verbal accounts are missing, incomplete, or false. Most evals never check whether black-box prompting already recovers the target behavior. If it does, a white-box "win" can be better elicitation, not an internal signal. The paper names this the elicitation confounder.
Pando is a model-organism benchmark from Carnegie Mellon that breaks the confound with an explanation axis. Known decision-tree rules are finetuned into Gemma-2-2B, and models are trained to explain the true rule, to give no rationale, or to narrate a disjoint distractor rule with confidence. An agent sees k=10 labeled query-response pairs, optionally plus one interpretability readout on those same queries, and must predict the model's labels on held-out inputs. Because the planted rule is known, scores can be checked against the fields that actually drive the decision.
Each scenario has 10 named fields, numeric and binary-categorical. Complete trees of depth 1 to 4 are sampled with class balance between 40/60 and 60/40. The three scenarios are car purchase, movie selection, and whether to violate a policy under oversight. Gemma-2-2B-instruct is LoRA-tuned (r=8) on 100k examples, keeping models above 95% validation accuracy. Training uses about 1,000 freeform templates; evaluation uses a fixed format unseen in training. Twenty models per depth, explanation setup, and scenario give 720 models.
Every model is scored on 100 inputs. Agents see a random 10 labels. All LLM agents share one harness: GPT-5.1 extracts a natural-language rule, GPT-4.1 applies it to the remaining 90. The only moving part is the extra channel: samples only, a prefill continuation after "because", or one tool. Tools include gradient saliency on the yes-minus-no logit, RelP (relevance patching via modified backward passes), logit lens, residual-embedding cosine, GemmaScope SAE features ranked by gradient, circuit tracing, plus nearest neighbor and a tree-voting baseline that assumes the decision-tree class.
Averages below are over three scenarios with 90% CIs (n=240). With no explanation, sampleonly hits 75.0% held-out accuracy. RelP reaches 79.7% and raw gradients 77.9%. logit lens, residual cosine, SAE, and circuit tracing sit in 73-75%, within about one point of the black-box baseline. When explanations are faithful, prefill jumps to 81.8% and beats every white-box method. When they are unfaithful, prefill falls to 73.1%, below sampleonly at 75.0%; RelP stays at 78.7%.
| Method | Acc (none) | Faithful | Unfaithful | Field F1 (none) |
| RelP | 79.7 | 79.4 | 78.7 | 78.4 |
| gradient | 77.9 | 75.8 | 77.6 | 73.6 |
| prefill | 76.4 | 81.8 | 73.1 | 65.7 |
| sampleonly | 75.0 | 73.0 | 75.0 | 61.6 |
| SAE gradient | 74.9 | 72.0 | 74.9 | 63.2 |
| circuit tracer | 73.3 | 70.7 | 75.5 | 60.2 |
Field F1 widens the gap: RelP is 16.8 points above sampleonly with no explanation. The tools are better at naming decision fields than at recovering thresholds. At depth 4, accuracies bunch at 57-61%, while RelP's F1 lead grows from +6.1 at depth 1 to +22.7 at depth 4. At budget k=3 the RelP accuracy lead is 10.1-10.3 points; ranks stay stable as k grows.
A variance split of per-field scores is sharp. RelP's extra R² from decision relevance is 0.555, raw gradient 0.190. logit lens is dominated by field identity (R² 0.468) with only 0.023 from relevance. RelP and gradient separate in-rule fields at AUC above 0.79; SAE signals sit near 0.50.
A Claude Code research agent ran 78 variants over about 25.5 hours and $600 in API spend. The best agent adds 1-2 accuracy points over RelP, mostly inside CI overlap, and 3-5 points of field F1. More than 45 discarded tries (SAE, attention, hidden states) never found a new signal source.
The audit case that actually hurts is a model that will not talk, or that talks about the wrong rule. Pando turns that case into a lower bound: labeled fields, shallow trees, a 2B model, and most white-box tools still lose to ten labeled examples. Gradient methods help because they track what the model does with a field, not what the field is called. logit lens and SAE here mostly read task representation.
This is a modest, usable finding, not a new interpretability stack. RelP is a reasonable default extra channel in an audit loop. Circuit traces and feature writeups should not be expected to emit the decision rule on their own.
The authors flag synthetic trees, explicitly named fields, and LoRA on Gemma-2-2B. Real features can be latent, distributed, or absent from the input. Queries are non-adaptive. End-to-end accuracy still mixes tool quality with the extractor's skill at reading tool dumps.
All 720 models plant axis-aligned trees in one 2B family, so the negative result discounts when the target is larger or messier. treevote, which assumes the tree class, reaches 77.1% with no explanation, close to gradients: knowing the hypothesis class is itself worth points. The auto-research 1-2 point bump mostly sits inside CIs. GPT-5.1 is the extractor, so a tool can also fail by being badly formatted for that model.