CHIVE: Evaluating LLM explanations via counterfactual experiments — common interpretability techniques show no uplift
a_karvonen · x · 2026-08-22
A new paper from the Anthropic Fellows Program, "Would this change your answer?", proposes evaluating explanations via counterfactual simulatability: whether an explanation helps predict model behavior on related counterfactual inputs.
The authors introduce CHIVE (Counterfactual Hypothesis Investigation Via Edits), an agentic pipeline that discovers unexpected model behaviors in the wild and investigates them with counterfactual prompt edits, yielding thousands of explanations with supporting counterfactual evidence. Two key findings:
- When evaluating common LLM interpretability techniques, they surprisingly find no uplift in an agent's ability to predict counterfactual model behavior from any technique studied.
- Using CHIVE investigations as training data — teaching models to predict outcomes of counterfactual experiments — generalizes to held-out OOD settings (e.g., detecting when a hint changed an answer), though results vary with training format.
More from Safety
- Vector Institute partners with Town of Caledon in its first municipal AI adoption deal — VectorInst · 2026-10-16
- 'Most dangerous technology in the world' put in toddler sandboxes, and we're surprised when it escapes — KevinNaughtonJr · 2026-10-16
- NYC Comptroller urges Anthropic to pause IPO until AI safety and governance concerns are addressed — ShakeelHashim · 2026-10-16
- Dwarkesh: Anthropic insider expects AI to match humans at AI research within 9 months — ShakeelHashim · 2026-10-16
- Anthropic and Gray Swan launch $300,000 indirect prompt injection red-teaming contest — bcherny · 2026-10-16
- Experts: AI's nearer bioweapons risk is amplifying capable actors, not enabling amateurs — HaydnBelfield · 2026-10-16