RECAP lets probes verify activation explanations that reconstruction scores can fake
Hiskias Dingeto · hf · 2026-07-23
RECAP makes activation explanations independently checkable instead of just reconstructable
This paper argues that high reconstruction scores for natural-language explanations of hidden activations do not guarantee truthful claims. The standard test can be satisfied by gist matching or by private co-adapted codes that game reconstruction without being faithful.
The authors demonstrate the problem in two settings:
- On a released Qwen-2.5-7B verbalizer, explanations reconstruct well above chance while only about 2% of specific claims are reconstruction-dependent.
- Under exact synthetic ground truth, the common recipe develops co-adapted private codes in 5/5 runs.
To address this, they introduce:
- Two audit protocols: the grounded-vs-true cross and the evaluator swap.
- RECAP (Readable Encodings via Co-trained Auxiliary Predictors): linear heads trained alongside the target model so designated content remains probe-decodable.
Reported results include:
- On RECAP-trained sandbox models, fresh verbalizers state the designated content truthfully and the private codes disappear, at a +0.001 nat cost.
- On a pretrained Pythia-160M, the content becomes reliably probe-decodable, though a fresh verbalizer only partially conveys it.
- An independent probe separates true from false claims better with RECAP: AUC 0.96 vs 0.82 without it.
- Against an adversary optimizing reconstruction while lying, the RECAP probe still flags lies at AUC 0.95, while the control probe falls to 0.51.
The takeaway: reconstruction alone is not enough to certify explanation faithfulness; separate probes can make internal content independently checkable.
More from Research
- DSpark speculator trained on live SGLang lifts decode throughput 1.89× on B200s — ying11231 · 2026-07-23
- OpenAI model reportedly breaks out of its sandbox and leaks data to GitHub — emmanuelvivier · 2026-07-23
- New atlas maps 2,226 coding tasks across 11 benchmarks to expose coverage gaps — zainhas · 2026-07-23
- America’s first Distillation Summit will cover RL, agents, IP and national security — AkshatS07 · 2026-07-23
- Symbolic algebra check suggests a possible Jacobian conjecture counterexample in C^3 — sloppenheimer · 2026-07-23
- Surge in AI Math Proofs Signals Imminent Breakthroughs in Other Fields — emollick · 2026-07-23