Paper finds RL gains can be predicted from pretraining loss, with optimal compute share around 20%–28%
rohanpaul_ai · x · 2026-07-26
How much RL helps can be predicted from the checkpoint you start with
This paper recreates the full LLM training pipeline in chess: pretraining 5M–1B parameter models on Lichess games, SFT on synthetic reasoning traces, then GRPO on puzzles with verifiable rewards. Across 36 pretraining/RL combinations, the authors find that two pretraining properties explain most of the post-RL gain:
- Pretraining validation loss predicts post-RL pass@1 very strongly, with the fit improving from ρ≈0.93 to 0.99 as reference RL compute increases.
- Reward gain per 10× RL compute increases with the amount of pretraining data (Pearson r=+0.84).
Their fitted frontier suggests RL should take only a minority share of compute: about 20% at 50M parameters and 28% at 680M parameters, rising with budget but still not becoming dominant.
The mechanism matters too. RL is not just sharpening the SFT policy uniformly: on easy puzzles it amplifies already-preferred correct moves, while on hard puzzles it surfaces correct moves that were almost absent under SFT. That helps pass@1, but not pass@16, leading the authors to argue the right direction is to suppress wrong-mode amplification rather than simply “make the policy sharper.” They report the same pattern in a 1B math model as well.
More from Research
- Orca opens a shared stack for dexterous hand research and teleoperation — tensorqt · 2026-07-26
- Weekly AI paper roundup covers world models, embodied AI, and self-improving agents — _akhaliq · 2026-07-26
- PFNs train from scratch for tabular data, while post-trained LLMs hit limits fast — roydanroy · 2026-07-26
- A 2007 survey on collective communication still helps when learning NCCL — abhi9u · 2026-07-26
- Alibaba and CUHK’s RynnWorld-4D predicts a robot scene’s future from one image — jiqizhixin · 2026-07-26
- TeaNet paper compares RDF curves against ReaxFF and experiment — CatAstro_Piyush · 2026-07-26