Goodfire Traces OLMo's Preference-Training Regression to Specific Dolci Pairs
allen_ai · x · 2026-09-10
Allen AI and interpretability firm Goodfire showed a concrete case: after preference training, OLMo gained general capabilities but became more likely to answer harmful requests framed as fiction or hypotheticals. Goodfire traced part of the regression to specific Dolci preference pairs, then tested targeted training changes and measured the fix without weakening other capabilities.
Allen AI's takeaway: publishing more than model weights matters — researchers need data and training details to trace unexpected behavior back to its source and verify fixes.
More from Safety
- UK AISI: under $80 of fine-tuning fully restores filtered bio capabilities in open-weight models — connoraxiotes · 2026-10-27
- Meta joins frontier labs reporting CoT controllability evals in Muse Spark 1.3 card — tomekkorbak · 2026-10-27
- Meta TBD underwent pre-launch red-teaming by 5 third-party safety firms — maksym_andr · 2026-10-27
- Is ChatGPT showing your chats to others? Screenshot sparks concern — chrisalbon · 2026-10-27
- Do canary tokens still work in the coding-agent era? Fake AWS keys may lure agents — kuza55 · 2026-10-27
- LLM bias judges name groups instead of saying 'unknown', refusing only 0.4% of the time — ZhijingJin · 2026-10-27