Anthropic study: interpretability tools fail to beat just reading transcripts

Research by @akarvonen and colleagues, done as part of the Anthropic Fellows Program (paper "Would this change your answer?"), proposes counterfactual simulatability as the standard for a good explanation: it should help predict model behavior in related situations, such as after renaming a variable or changing wording. They built an evaluation set of thousands of real behaviors and the CHIVE pipeline (described in an Anthropic blog as an automated agent pipeline for discovering unexpected real-world LLM behaviors and explaining them via counterfactual prompt edits), runnable on any model or prompt source. The headline finding: existing interpretability tools do not, on average, provide this predictive power.

Confirmed

• The study tested three activation-based tools that had succeeded in prior auditing games—activation oracles, natural-language autoencoders (NLAs), and SAEs—on diagnosing agent failure causes and predicting prompt edits, against a baseline of just reading the transcript.
• None of the tools beat reading the text, robust across hyperparameters, prompts, and Fable target loops.
• Behavioral causes vary wildly: specific words in prompts, model quirks (e.g., misremembered actor facts), or abstract attributes like an angry user tone, making automated diagnosis very hard.
• Pipeline examples: Gemma wrote buggy code due to misleading variable names; Qwen3-8B tended to pick the first list item then confabulate justifications, exposing unfaithful chain-of-thought.
• @akarvonen notes NLAs are hard to use and misleading in practice: a 500-token transcript can yield up to 50k tokens of output that rarely answers questions directly; NLAs are easier on fine-tuned model organisms.
• On why tools helped in game audits but fail here: in audits, out-of-transcript "banana" mentions straightforwardly imply a model organism, whereas in real behavior analysis unsupervised outputs rarely answer the question and come with thousands of tokens of noise requiring tea-leaf-reading.
• The team will use findings as training data to teach models to explain their own behavior, generalizing to held-out OOD evaluations (e.g., detecting whether a suggestion in a prompt changed the answer), with effects varying by training format; the pipeline also exposes unfaithfulness in the wild.
• @aryaman2020 cited the paper "Pando": a new model-organism benchmark eliminating "elicitation confounds," tested on 720 fine-tuned models; when models refuse to explain themselves, black-box prompting often beats white-box tools.

Unconfirmed

• @aryaman2020 was surprised by the NLA results; discussants proposed simple token-level gradient attribution, citing related work, but no experimental conclusions yet.

Why it matters

• If white-box tool gains come mainly from better elicitation rather than true internal signal, interpretability evaluation paradigms need rethinking.
• In practical agent diagnosis, simple baselines (reading transcripts) remain the bar to beat; tool development should be measured against them.
• Counterfactual simulatability offers an operational standard for explanation quality, and turning investigations into training data shows a path for interpretability to feed back into model self-explanation.

2026-08-22 ~ 2026-08-22 · 15 related posts

• On Model Interpretability: Good Explanations Should Predict Related Situations — a_karvonen · 2026-08-22
• Study Finds Interpretability Tools Fail to Predict Model Behavior Effectively — a_karvonen · 2026-08-22
• [source] CHIVE Pipeline Reveals Gemma Coding Errors Due to Misleading Variable Names — a_karvonen · 2026-08-22
• Model Behavior Causes Vary: From Specific Words to Abstract Tone — a_karvonen · 2026-08-22
• Evaluating Agent Diagnostics: Do Interpretability Tools Help Over Reading Transcripts? — a_karvonen · 2026-08-22
• [source] Interpretability Tools Fail to Beat Reading Transcripts in Agent Diagnostics — a_karvonen · 2026-08-22
• Qwen3-8B Case Study: Picking First Option Then Fabricating Reasoning — a_karvonen · 2026-08-22
• Anthropic turns interp investigations into training data so models explain their own behavior — a_karvonen · 2026-08-22
• [source] CHIVE: Evaluating LLM explanations via counterfactual experiments — common interpretability techniques show no uplift — a_karvonen · 2026-08-22
• Study: Interpretability Tools Provide No Uplift for In-The-Wild Behavior Evals — a_karvonen · 2026-08-22
• Paper questions interpretability: Black-box prompting often beats white-box tools — aryaman2020 · 2026-08-22
• NLA results surprising, token-level attribution may help — aryaman2020 · 2026-08-22
• Study finds Neural Language Activations difficult to use in practice for interpretability — a_karvonen · 2026-08-22
• Anthropic evaluates LLM behavior explanations via counterfactual experiments — saprmarks · 2026-08-22
• Study Finds Interpretability Tools Largely Fail to Predict 'In the Wild' Behaviors — saprmarks · 2026-08-22