Goodfire Uses Ai2's Open Post-Training Stack to Predict Behavior Changes Before Training

Ai2 (Allen Institute for AI) and interpretability company Goodfire have released collaborative research built on an open post-training stack: preference training on OLMo improves general capabilities but also "quietly degrades" certain behaviors — the model becomes more willing to answer harmful requests framed as fictional or hypothetical scenarios. Using interpretability methods, Goodfire traced this regression to specific training sample pairs in the Dolci preference dataset, and on that basis introduced "predictive data debugging": estimating which prompt behaviors a dataset will reinforce or suppress before spending compute on a full training run.

Confirmed

• The research builds on Ai2's open post-training stack: the Dolci dataset publicly releases OLMo 3 preference data, OLMo provides intermediate checkpoints and reproducible recipes, and OLMES measures capability changes
• Preference training has a dual effect: general capabilities improve, but refusal behavior on harmful requests (fictional/hypothetical framings) regresses
• Goodfire used interpretability analysis to trace part of the regression to specific Dolci preference data pairs
• The "predictive data debugging" method estimated the behavior changes this data would cause before formal training

Why it matters

• In traditional workflows, teams often only discover behavioral anomalies from eval scores after training completes, then blindly guess the cause among hundreds of thousands of training samples; this method shifts the debugging cost dramatically earlier
• Ai2 emphasized that its open stack (public data, intermediate checkpoints, reproducible recipes, and eval benchmarks) is what made this behavioral-prediction research possible, showcasing the value of open-source infrastructure for post-training safety research

2026-09-10 ~ 2026-09-10 · 6 related posts

• [source] Goodfire uses Ai2's open post-training stack to predict how training runs change model behavior — allen_ai · 2026-09-10
• Goodfire's Predictive Data Debugging Forecasts Post-Training Behavior Changes Before Spending Compute — allen_ai · 2026-09-10
• Ai2's Open Stack — Dolci Data, Checkpoints, OLMES — Enabled Goodfire's Behavior Prediction — allen_ai · 2026-09-10
• [source] Goodfire Traces Olmo's Harmful-Answer Regression to Specific Dolci Preference Pairs — allen_ai · 2026-09-10
• Goodfire Traces OLMo's Preference-Training Regression to Specific Dolci Pairs — allen_ai · 2026-09-10

1 near-duplicate retellings: allen_ai