Unified training: one maximum-likelihood algorithm for pretraining through RL
adityakusupati · x · 2026-10-31
Researchers propose "unified training," a single algorithm spanning the entire foundation model pipeline:
- Training has historically been staged—pretraining, midtraining, SFT, RLHF, RLVR—each with its own algorithms (cross-entropy to GRPO) optimized in isolation.
- The authors show all roads lead to maximum likelihood: pretraining, midtraining and SFT already are ML; the seeming exception, RL, is covered by their earlier MaxRL framework, extensible to REINFORCE and other RL objectives.
- Instead of stages, one algorithm plus an evolving data curriculum: mix data sources and adjust proportions over training.
Payoffs: simpler infrastructure and a framework for better science—reframing the question from "which algorithm" to "what data, how much, when."
More from Research
- Correction: newer model gets worse on the extreme-hard tail, pass@5 crossover predicted — teortaxesTex · 2026-10-31
- Meta demos titanRL, TorchTitan's RL library for LLMs, teaching Qwen Connect 4 — ezyang · 2026-10-31
- Biohub's Remo-ID spots sepsis from blood cell images, 43/47 correct in early study — alex_peys · 2026-10-31
- 240-Year-Old Voting Paradox Gets a Shape: It's a Klein Bottle — stevenstrogatz · 2026-10-31
- OpenAI releases 700+ math manuscripts in one batch; Fortnow says it beats 30 years of progress — kimmonismus · 2026-10-31
- DeepMind's Economics of Intelligence: AI reshapes economy via agent coordination, not solo model smarts — Scobleizer · 2026-10-31