Anthropic Fellows release CHIVE: training models to explain their own behavior, with generalization

Adam Karvonen's team (with Euan Ong, Subhash Kantamneni and Samuel Marks, done during the Anthropic Fellows Program) released the CHIVE paper and blog post (arXiv:2608.16747), a method for training models to explain their own behavior, with self-explanation generalizing to evaluations it was never trained for. John Schulman reshared it approvingly.

Confirmed

• Motivation: let models explain their own behavior, such as why they ignore user requests or write buggy code.
• Method: the CHIVE pipeline automatically finds anomalous in-the-wild behaviors and runs counterfactual editing experiments to generate thousands of high-quality explanations grounded in the model's real behavior, forming a single general dataset for training.
• Results: trained models generalize on held-out evaluations, including the hint evaluation ("would removing the hint change your answer?").
• Generalization: prior self-explanation training generalized narrowly because datasets were narrow; this work never trained on hints yet improved hint evaluations.
• Cross-model training: training Qwen on investigation data about Llama's behavior (and vice versa) performed comparably to self-training, so no privileged self-access is needed.
• Limitation: the authors acknowledge open-ended self-explanation remains unreliable.

Unconfirmed

• gerardsans disputed the framing, arguing it misleadingly suggests "models have a self" when they are stateless inference with no inner self to explain; the disagreement remains unresolved.

Why it matters

• The work moves interpretability from external analysis to trainable self-explanation grounded in real behavior with measurable metrics, potentially a practical tool for safety auditing and behavior diagnosis; the naming controversy also highlights the need to carefully delineate such capabilities from claims about model "selves".

2026-09-05 ~ 2026-09-05 · 6 related posts

• [source] Anthropic Fellows train models to explain their own wild behaviors with generalization to held-out evals — a_karvonen · 2026-09-05
• Self-explanation training generalizes beyond narrow hint formats to held-out evals — a_karvonen · 2026-09-05
• Cross-model training matches self-training: Qwen learns from Llama investigations without privileged access — a_karvonen · 2026-09-05
• [source] CHIVE paper: evaluating LLM behavior explanations in the wild with counterfactual experiments — a_karvonen · 2026-09-05
• Training models to explain their own behavior: new interpretability dataset generalizes to held-out evals — a_karvonen · 2026-09-05
• "The system has no self": critic says behavioral-explanation training misunderstands stateless LLM inference — gerardsans · 2026-09-05