Understanding Reasoning from Pretraining to Post-Training
Jingyan Shen, Ang Li, Salman Rahman, Yifan Sun, Micah Goldblum, Matus Telgarsky, Pavel Izmailov
cs.LG, cs.AI, cs.CL
2026-07-18
Pretraining 5M-to-1B models on human chess games, then SFT and GRPO, the authors find post-RL performance at fixed RL compute is predicted by pretraining loss and the RL reward slope rises roughly linearly with pretraining tokens.
Reinforcement learning has become central to improving complex reasoning in large models, but RL post-training is largely studied in isolation from the pretraining before it. Two basic questions stay open: how pretraining choices (size, data) shape the returns to RL compute, and what RL actually does to the model. The standard LLM setting cannot answer these cleanly: corpora are vast and uncontrolled, behaviors are hard to attribute to pretraining versus RL, and systematic compute sweeps across both stages are prohibitively expensive.
This paper uses chess as a controlled testbed. Chess has a small action space (an 81-token vocabulary), exact engine verification, fine-grained data quality control, and specialized models reach nontrivial performance at affordable compute, all suited to systematic sweeps.
The pipeline mirrors standard LLM training. Pretraining trains 5M-to-1B models (Qwen3 dense architecture) on 54B tokens of Lichess human Blitz and Rapid games (2022, at least 10 plies, Elo 800-3000). SFT fine-tunes on synthetic reasoning traces, where K continuations are merged into a tree, serialized depth-first by shared prefix, and the model learns to pick the best line. RL uses GRPO on 156K chess puzzles with binary reward, paying 1 only if every move matches the ground-truth line. The test set is 1,480 tactical puzzles across Elo bins B1 to B4.
The paper fits a joint scaling law: post-RL pass@1 at fixed RL compute is strongly predicted by pretraining validation loss (Spearman absolute correlation 0.93 to 0.99), and the local slope measuring reward per decade of RL compute correlates roughly linearly with pretraining tokens (Pearson r = 0.84). In short, more pretraining makes RL more worthwhile.
What RL does is not a uniform sharpening of the SFT policy (which would amount to scaling temperature). On easy puzzles (B1-B2) it amplifies correct moves the SFT policy already preferred; on hard puzzles (B3-B5) it promotes correct moves from the low-probability tail (epsilon 0.05) into the top-k, though it also amplifies wrong moves. On chain-of-thought, the model widens search rather than deepening it: the width-to-depth ratio and branching factor rise while maximum search depth stays flat. It gets stronger on continuations up to five moves but still struggles on longer ones, meaning RL improves candidate generation and selection faster than long-horizon search.
The pattern transfers to math: a 1B OLMo-2 pretrained on 200B tokens (70% math corpus), with 14 checkpoints from 10B to 200B tokens, SFT and then RL on GSM8K and MATH, reproduces the same regularity: lower pretraining loss yields higher post-RL performance, with the slope rising linearly in log tokens.
On the compute-optimal frontier, the optimal RL share rises with total compute: about 20% at 50M parameters up to 28% at 680M (discussions cite 30%), while pretraining token allocation still roughly follows Chinchilla. At low compute, RL is strongly initialization-limited.
For pretraining and post-training teams this offers a quantifiable interface: pretraining loss is a strong predictor of RL returns, and the optimal RL compute share rises with scale rather than holding fixed. That challenges the common view of pretraining and RL as separate stages and suggests jointly optimizing both when allocating total compute. The finding that RL widens rather than deepens search also explains why models stay weak on long-horizon reasoning.
Honestly, the conclusion's reach is narrow: models top out at 1B, chess differs sharply from natural language (small vocabulary, exact verification, no entangled world knowledge), and the RL uses unique-solution binary rewards unlike the partial-credit, open-ended rewards of language tasks. Extrapolating directly to frontier-scale models needs care.
The authors are explicit: chess is not natural language, with reasoning not entangled with world knowledge or fluency; puzzles have unique designated solutions and binary rewards, more restricted than language-task partial credit; models reach only 1B parameters, so the scaling trends may differ at larger scale; the tree-based CoT format is specific, and other formats could yield different dynamics; and the RL slope association is a local empirical trend over the studied compute range where benchmarks are not saturated, not a global relationship. These boundaries are stated clearly and are the honest part of the paper.