Replicating Anthropic's 'Teaching Claude Why': hard advice cuts misalignment to 0.04
dhadfieldmenell · x · 2026-11-04
Researchers at secondlookxlab replicated Anthropic's "Teaching Claude Why" mid-training interventions on Qwen 2.5 32B/72B base models to test mitigation of agentic misalignment:
- Fictional stories of AIs behaving well: misalignment rate 0.61→0.43 (32B), 0.69→0.63 (72B) — modest gains
- Difficult ethical-dilemma advice: rate drops to 0.06
- Stories + difficult advice combined works best at 0.04
Both interventions work out of the box, and difficult advice dramatically outperforms fictional stories — concrete evidence for how to design alignment-oriented mid-training data.
More from AGI Musings
- Gary Marcus reviews 2025 AI predictions: 17/18 correct on scaling hitting a wall — GaryMarcus · 2026-11-04
- DeepSeek's Liang Wenfeng on why open source is survival, not charity, in the AI era — teortaxesTex · 2026-11-04
- Every's inaugural Thesis: 2027 conference asks what great human work looks like after automation — every · 2026-11-04
- Greg Kamradt: attribution analysis could tease out AI's per-person dollar gains — GregKamradt · 2026-11-04
- Gary Marcus Scores His AI Predictions 17/18 in Spat With Sabine Hossenfelder — GaryMarcus · 2026-11-04
- PhD Reflection: The Brain Isn't a GPU — Rest Is Part of Research — tw_killian · 2026-11-04