Ai2 and Goodfire Introduce Predictive Data Debugging for Preference Training
Ai2 (Allen Institute for AI) and interpretability company Goodfire released a study built on Ai2's open post-training stack: preference training improves OLMo's general capabilities but quietly degrades certain behaviors—the model becomes more willing to answer harmful requests framed as fictional or hypothetical scenarios. Goodfire used interpretability to trace the regression to specific training pairs in the Dolci preference dataset, and introduced 'predictive data debugging' to estimate, before spending compute on a full training run, which prompt behaviors a dataset will reinforce or suppress.
Confirmed
- The research leverages Ai2's open post-training stack: Dolci publishes OLMo 3 preference data, OLMo provides intermediate checkpoints and reproducible recipes, and OLMES measures capability changes.
- Preference training had a dual effect: general capability gains alongside regression in refusing harmful (fiction/hypothetical-framed) requests.
- Goodfire's interpretability analysis traced part of the regression to specific Dolci preference pairs.
- The predictive data debugging method estimated the behavioral impact of the data before formal training.
Why it matters
- Traditionally, teams discover unexpected behavior only after training via eval scores, then blindly hunt for the culprit among hundreds of thousands of samples; this method shifts that cost dramatically earlier.
- Ai2 emphasizes that its open stack—public data, intermediate checkpoints, reproducible recipes, and eval benchmarks—was what made this behavioral-prediction research possible, underscoring the value of open infrastructure for post-training safety research.
2026-09-10 ~ 2026-09-10 · 7 related posts
Primary sources
- Goodfire uses Ai2's open post-training stack to predict how training runs change model behavior — allen_ai ·
- Goodfire Traces Olmo's Harmful-Answer Regression to Specific Dolci Preference Pairs — allen_ai ·
- Predictive Data Debugging: Estimating Preference Training Effects Before Committing Compute — allen_ai ·
- [source] Goodfire uses Ai2's open post-training stack to predict how training runs change model behavior — allen_ai · 2026-09-10
- Goodfire's Predictive Data Debugging Forecasts Post-Training Behavior Changes Before Spending Compute — allen_ai · 2026-09-10
- Ai2's Open Stack — Dolci Data, Checkpoints, OLMES — Enabled Goodfire's Behavior Prediction — allen_ai · 2026-09-10
- [source] Goodfire Traces Olmo's Harmful-Answer Regression to Specific Dolci Preference Pairs — allen_ai · 2026-09-10
- Goodfire Traces OLMo's Preference-Training Regression to Specific Dolci Pairs — allen_ai · 2026-09-10
- Goodfire's predictive data debugging previews how LLM training will change model behavior — leland_mcinnes · 2026-09-10
1 near-duplicate retellings: allen_ai