Training models to explain their own behavior: new interpretability dataset generalizes to held-out evals
a_karvonen · x · 2026-09-05
- Adam Karvonen shares new interpretability work: models trained on thousands of explanations of their own in-the-wild behaviors (e.g. why they ignored a request or made a coding error) generalize to held-out evals from a single general dataset.
- John Schulman endorses the direction: a metric for explanation quality enables hillclimbing, and counterfactual simulatability seems sound. The dataset+pipeline produces more diverse and realistic test cases than prior work, and models can also be trained to write better post-hoc explanations.
More from Research
- Three reasons AI is not conscious and may never be, per Nature essay — anilkseth · 2026-10-21
- Aviation tracks every part's provenance — agents can reason over that graph — colinmcnamara · 2026-10-21
- SFT details: instruction/reasoning/code/math mix, synthetic + human traces, no epoching — TimDarcet · 2026-10-21
- Mid-training recipe disclosed: 32k→131k context extension, dropping logit distillation for tokens — TimDarcet · 2026-10-21
- Mathematician Daniel Litt concedes his AI bet, but says AI won't solve math — ziv_ravid · 2026-10-21
- AI drug discovery reality check: 'give me everything that binds,' then filter for toxicity — burny_tech · 2026-10-21