CMU paper shows pretraining, SFT and RL can run as one unified likelihood-based algorithm
omarsar0 · x · 2026-11-01
A CMU-led paper (with JHU and Aurora Innovation) argues the staged pipeline of pretraining, midtraining, SFT and RL is not fundamental. By deriving suitable RL objectives as maximum-likelihood fitting to model-generated targets, all stages share one loss and one trainer — dubbed unified training. Design reduces to choosing which data, in what proportions and when; staged training becomes a special case. A 124M model trained from scratch on reasoning puzzles beat staged training on RL tasks and generalized to larger unseen puzzles, reusing the midtraining corpus up to 4 times.
More from Models
- Newest LLM isn't automatically best for business: reliability and cost beat benchmarks — DavidLinthicum · 2026-11-01
- Using AI to crack a spatial folding puzzle on a Punjabi TV quiz show — satnam6502 · 2026-11-01
- Friends resubscribed to ChatGPT for 'unlimited' use — worst timing ever — Angaisb_ · 2026-11-01
- GPT-6.1 Terra ranks #14 on Toloka Arena as pass^5 falls to 59% from ~72% — MParakhin · 2026-11-01
- OpenAI employee: GPT-6.1 Sol feels better than GPT-6 Astra, nearly as smart and much faster — willdepue · 2026-11-01
- What happens to devs if Chinese firms stop releasing free open-source models? — blondewalker · 2026-11-01