Anthropic Fellows Train LLMs to Explain Their Own Behavior, Generalizing to Unseen Evals

Adam Karvonen's team (work done during the Anthropic Fellows Program, with co-authors Euan Ong, Subhash Kantamneni, and Samuel Marks) released the CHIVE paper and blog post (arXiv:2608.16747), introducing a new method for having models explain their own behavior and demonstrating that trained self-explanation ability generalizes to evals it was never trained on. John Schulman reshared it approvingly.

Confirmed

• Motivation: enable models to explain their own behavior, such as why they ignored a user request or wrote buggy code.
• Method: use the CHIVE pipeline to automatically discover anomalous model behaviors in the wild, then generate thousands of high-quality explanations grounded in the model's actual behavior via counterfactual editing experiments, forming a single general-purpose dataset for training.
• Results: the trained model generalized on held-out evals, including the hint eval ("would removing the hint change your answer?") and other benchmarks.
• Generalization breakthrough: the authors note that past self-explanation training generalized very narrowly (e.g., only across a single hint format) because the datasets were too narrow; this study never trained on hints yet still improved hint eval performance.
• Cross-model training: training Qwen on investigation data about Llama's behavior (and vice versa) worked about as well as self-training, showing no privileged access to the model's own behavior is needed.
• Limitations: the authors acknowledge that open-ended self-explanation remains unreliable for now.

Why it matters

• The approach advances interpretability from external analysis to models being trained to explain themselves, with explanations grounded in real behavior rather than confabulation, paired with measurable metrics—potentially a practical tool for model safety audits and behavioral diagnostics.

2026-09-05 ~ 2026-09-05 · 5 related posts

• [source] Anthropic Fellows train models to explain their own wild behaviors with generalization to held-out evals — a_karvonen · 2026-09-05
• [source] Self-explanation training generalizes beyond narrow hint formats to held-out evals — a_karvonen · 2026-09-05
• Cross-model training matches self-training: Qwen learns from Llama investigations without privileged access — a_karvonen · 2026-09-05
• [source] CHIVE paper: evaluating LLM behavior explanations in the wild with counterfactual experiments — a_karvonen · 2026-09-05
• Training models to explain their own behavior: new interpretability dataset generalizes to held-out evals — a_karvonen · 2026-09-05