CMU paper unifies pretraining, SFT and RL into one likelihood-based algorithm

burny_tech · x · 2026-11-05

A CMU/JHU/Aurora Innovation paper proposes Unified Training, arguing the staged foundation-model pipeline (pretraining, midtraining, SFT, RL) is not fundamentally separable. Key points:

RL objectives are derived as maximum-likelihood fitting to model-generated target distributions, so all stages share one likelihood-based objective
The training lifecycle reduces to choosing which data to train on, in what proportions, and when; pretraining data, demonstrations, and successful RL generations mix in a single trainer
Conventional staged training becomes a special case

In 124M-parameter experiments, the best unified-training configs beat staged training on reasoning accuracy and generalization to harder unseen problems while better preserving prior capabilities, and provide a stronger starting point for continual learning.

Original post →

CMU paper unifies pretraining, SFT and RL into one likelihood-based algorithm

More from Infra