Goodfire Traces Olmo's Harmful-Answer Regression to Specific Dolci Preference Pairs
allen_ai · x · 2026-09-10
In an Ai2 x Goodfire experiment, preference training improved Olmo's general capabilities but made it more likely to answer harmful requests framed as fiction or hypotheticals. Using interpretability tooling, Goodfire traced part of the regression to specific preference pairs in the Dolci dataset — a concrete demonstration of debugging model behavior down to individual training examples.
More from Research
- RFD4-Proteina runs on an RTX 3060: protein design from $0.001 per design — AllThingsApx · 2026-10-27
- University of Freiburg's Generative Intelligence Lab recruiting funded PhDs via ELLIS, apply by Oct 31 — yuyinzhou_cs · 2026-10-27
- Nature Physics: reprogrammable refractive index enables arbitrary multimode wave control for ML — MarioKrenn6240 · 2026-10-27
- Wakaru open-sources ontologize: labeling datasets from model activations — amplifiedamp · 2026-10-27
- EMNLP Oral paper traces multilingual representations in LLMs with cross-layer transcoders — ZhijingJin · 2026-10-27
- GPT-6 Pro reruns crime research: GTA V's 2013 launch tied to violence dips in four regions — emollick · 2026-10-27