How to train your model organism: paper finds DPO beats SFT, training recipes shape interpretability conclusions
davidbau · x · 2026-10-15
- Xilin Wang, David Bau and Byron C. Wallace (Northeastern) release "How to train your model organism" (arXiv 2610.10203), arguing that single-objective training of alignment model organisms (backdoors, sycophancy, spurious correlations) is insufficient for whitebox interpretability research.
- They propose validating organisms on three objectives: target-behavior installation, general-capability preservation, and output naturalness (CoT and activations).
- Re-auditing two public organism suites shows chat quality and CoT naturalness degrade substantially across recipes, and validation metrics predict how well interpretability methods recover installed behavior (e.g., logit lens readouts covary with general capabilities).
- They introduce a model-merging-based multi-objective training approach and a new clinical-bias organism suite. Key recommendations: prefer DPO to SFT, don't mix in chat data, and merge back toward the base model.
- Paper, code and blog post are publicly released.
Related event: David Bau's Team Publishes Guide on Training Model Organisms(2 posts)→
More from AGI Musings
- AI cost falling 54x faster than electricity ever did, five years in — LinusEkenstam · 2026-10-15
- Herbert Simon's 40-year-old advice on computers nails how to use AI today — msg · 2026-10-15
- AI researcher Thomas Dietterich debunks multi-agent analogy: 1000 copies of me can't build a toaster — tdietterich · 2026-10-15
- Scholar's long post on Anthropic's societal-challenges push: earnest aims, ecosystem worries — sethlazar · 2026-10-15
- AI Auditing Pioneer Deb Raji Joins Stanford HAI to Tackle AI Harm Accountability — StanfordHAI · 2026-10-15
- Will the frontier-lab and closed-vs-open capability gap widen in six months? — panickssery · 2026-10-15