Paper questions interpretability: Black-box prompting often beats white-box tools
aryaman2020 · x · 2026-08-22
The paper "Pando" investigates if interpretability methods work when models won't explain themselves, introducing a benchmark to control for the "elicitation confounder."
Key Findings:
- Across 720 finetuned models, black-box prompting matches or exceeds white-box methods when explanations are faithful.
- When explanations are absent/misleading, gradient-based attribution improves accuracy by 3-5%, with Relevance Patching (RelP) offering the largest gains.
- Logit Lens, Sparse Autoencoders (SAE), and circuit tracing provided no reliable benefit.
- Gradients track decision computation, while other readouts are dominated by task representation.
More from Research
- Five pharma giants federally train OpenFold3 on 20,000 private structures, +11 pts — jiqizhixin · 2026-10-16
- SWAN: a semantic-level text watermark that survives paraphrasing — burkov · 2026-10-16
- Interp researchers push back: 'networks are already clean' claim rests on dropout artifact — aryaman2020 · 2026-10-16
- LayerTrim: training-free pruning keeps reasoning LLMs intact on AIME — SeoulNatlUniv · 2026-10-16
- Simular presents neuro-symbolic computer use agents at BayLearn — xwang_lk · 2026-10-16
- Google's Hartmut Neven disputes chip test, outlines plan to link a brain to a quantum computer — Chris_Armstrong · 2026-10-16