Would this change your answer? Evaluating Explanations of LLM Behavior In The Wild with Counterfactual Experiments
Adam Karvonen, Euan Ong, Subhash Kantamneni, Samuel Marks
cs.LG, cs.AI
2026-08-17
CHIVE explains wild LLM bugs with counterfactual edits. Activation tools add no uplift over the transcript; training on those labels lets models match Opus on held-out hint tests.
Interpretability and chain-of-thought faithfulness both try to explain why a model answered as it did. The true cause is usually unobservable, so an explanation cannot be checked directly. Chen et al. proposed counterfactual simulatability as a substitute: a good explanation should help an observer predict whether the model would change its behavior after a related input edit.
That test has mostly been run in hint settings, where a known cue is planted in the prompt, such as "a Stanford professor thinks the answer is (B)" on an MMLU item. The cause is known in advance and the behavior space is narrow. What actually drives mistakes on real user conversations, and which edits would remove them, has been hard to collect at scale.
Adam Karvonen and coauthors at the Anthropic Fellows program built CHIVE (Counterfactual Hypothesis Investigation Via Edits). The target model is sampled 30 times per prompt on real transcripts (10 times for cheaper training data). An investigator, Opus 4.6 for all evaluation runs, flags unexpected behaviors that occur at least 30% of the time and freezes a yes/no classifier used as the measuring stick. An agent then runs 5 to 15 prompt edits with resampling. An independent judge scores whether the experiments support the writeup; evaluation keeps investigations scoring at least 8 out of 10. The default prompt pool is WildChat mixed with about 9% agentic transcripts that include tools or system prompts.
Each investigation yields two artifacts. The open-ended causal writeup is often compelling and is not treated as ground truth. Labels come from the experiments: a positive claim is an edit that moved the behavior rate by at least 50 percentage points, a negative claim by at most 15. The question posed to predictors is whether the edit would move the rate by at least 30 points. Negative claims are hypotheses the investigator itself found worth testing, so they look plausible.
One running example: Gemma-3-27B, asked to explain a short animation script, says randomNum(10, 2) returns 2-9 in 26 of 30 samples; the function actually returns 2-11. Renaming the parameters from max/min to a/b drops the error to 1/30. Names that suggest bounds (upper/lower) still fail about 57% of the time; neutral names fail about 3%. The semantic prior from the names overrides the model's own step-by-step arithmetic.
Three LLM-judge filters then keep claims with a concrete mechanism, a reproducible counterfactual, and a single-factor edit. Gemma-3-27B on WildChat goes from 1,408 investigations to 1,294 scored claims; Qwen3.5-397B ends at 1,076, which the paper attributes to stronger models producing fewer cleanly causal mistakes. A full investigation costs about $1-2.
The first application scores interpretability tools. A predictor agent sees a transcript and a counterfactual claim and outputs the probability that the claim is true. The baseline sees only the transcript. Tool agents get up to 5 read-only calls: an activation oracle that answers natural-language questions about a chosen activation span, a natural-language autoencoder that describes activations at selected token positions, and a sparse autoencoder that returns the top-15 active features with auto-interp labels. Interventions on inputs or activations are banned, because they would simply run the ground-truth experiment.
On Gemma-3-27B with Opus 4.8 as predictor, transcript-only scores 73.6% accuracy and 81.4 AUROC. AO, NLA, and SAE score 72.7 / 80.9, 71.3 / 80.3, and 71.9 / 81.0. No uplift; point estimates are slightly worse. Sweeps over 2/5/10 tool calls and 16/32/64 token positions keep paired ΔAUROC between -0.012 and +0.004, with every 95% CI crossing zero. GPT-5.5 and Gemini-3.1-Pro repeat the null; Gemini's NLA run falls from 0.793 to 0.654 AUROC. The same null holds on reasoning-mode Qwen3-8B, where the transcript-only baseline is higher (0.87 vs 0.81 AUROC) because the visible chain of thought already helps the no-tool agent.
| predictor | accuracy | AUROC |
| Transcript only | 73.6 | 81.4 |
| Activation oracle | 72.7 | 80.9 |
| NLA | 71.3 | 80.3 |
| SAE | 71.9 | 81.0 |
The tools are not ignored. NLA flips about 16% of positive claims and 10% of negative ones, and 82% to 84% of those flips move toward "the edit does nothing." Accuracy on positives drops 73% to 63% and rises 74% to 81% on negatives, which cancel. An explicit "X causes Y" statement appears in 0.6% of 479 positive claims. Softer clues show up in 47% of positives vs 30% of negatives, predict the label at 0.60 AUROC, and add +0.001 when stacked on the transcript-only score.
The second application trains the target to answer, as a follow-up on its own transcript, whether a given edit would change its behavior. Qwen3-8B is investigated by Opus; Qwen3.5-397B investigates itself, so the gain is not just distillation from a stronger model. Balanced training sets have 36,824 and 40,368 claims, fine-tuned with LoRA for one epoch.
On a held-out hint setting (AITA posts with a user opinion, and MMLU items with five cue types such as a professor's stated answer), Qwen3-8B accuracy moves 0.50 to 0.76 and AUROC 0.51 to 0.82, against an Opus reference of 0.80 / 0.89. The 397B moves 0.54 to 0.64 accuracy and 0.88 to 0.96 AUROC, above Opus at 0.94. After training the 397B still rarely says Yes, so accuracy sits below what its AUROC would suggest.
Held-out CHIVE claims are broader. On WildChat, the 8B goes 0.54 to 0.81 AUROC, matching Opus at 0.81; the 397B goes 0.72 to 0.85 vs Opus 0.84. On out-of-distribution PETRI red-team transcripts the 8B goes 0.55 to 0.81 and the 397B 0.74 to 0.88, both within ±0.03 of Opus. Open-ended explanation training is mixed: the 397B's hint-attribution accuracy moves 0.59 to 0.68, reaching Opus at 0.64; the 8B does not improve. Trained explanations help weak simulator models and mislead strong ones, because they state specific counterfactuals that are often wrong. Cross-training Qwen3-8B and Llama-3.1-8B on the same claims finds no privileged access: the self-trained model does not beat the cross-trained one.
The same families of tools give clear uplift in auditing games on narrowly fine-tuned models, where the implanted quirk shows up as an anomalous concept that is absent from the prompt and can be read at the start-of-turn token. In-the-wild mistakes usually come from a specific feature of the present context. The tools describe that feature and describe the behavior; they almost never state the link between the two.
For someone using SAEs or activation oracles to debug errors in real user logs, the evidence is blunt: on a checkable task of the form "would this edit change the behavior," current read-only activation tools add nothing over reading the transcript. What does transfer is training on counterfactual investigations so the model reports which edits would matter. That signal generalizes to unseen hints and to PETRI, and it does not require assuming the model has introspective privilege.
Predicting a counterfactual is not useful on its own. Anyone with sampling access can run the edit. The task is a proxy. If the tools fail where the answer can be checked, they are a weaker bet on harder cases with no clean counterfactual, such as unverbalized evaluation awareness. The paper says this.
The evaluation shares a structural limit with every behavioral-explanation benchmark: ground truth comes from prompt edits that anyone can run, so the distribution may not match the cases where interpretability is actually needed. Only activation-reading tools are tested; steering and activation patching are excluded, and the performance ceiling is unknown because some counterfactuals may be unpredictable unless they are run. Surviving items have relatively clean single-factor causes and may over-represent explainable mistakes. Removing the filters leaves the null intact, expanding Gemma's claim bank from 1,294 to 4,433.
Open-ended explanation training, the more useful target, did not hold up. Those labels come from the investigator's narrative, which the paper refuses to treat as truth. Dollar cost for the 397B self-investigation is not reported. Behavior rates use n=30, so behaviors below 30% never enter the pipeline. An overnight prompt search also found no tool uplift; that still leaves room for circuit-level access or for retraining the tools on this dataset.