Demystifying Reinforcement Learning Post-Training of Language Models
Donovan Clay, Saket Gollapudi, Sankar Harilal, Min Jang, Jacob Morrison, Sewoong Oh, Natasha Jaques
cs.LG, cs.AI, cs.CL
2026-08-25
A UW and Ai2 sandbox of RLVR finds sparse rewards only amplify covered behaviors, while a process reward lifts Qwen2.5-7B on one AIME item from 10.2% to 92.2%.
RL with verifiable rewards (RLVR) is how many labs now lift math and code. Two results in the literature sit badly together. Shao et al. showed that Qwen math models can still gain on math benches when the reward is random or even wrong, which made some people treat post-training success as a base-model artifact. Yue et al. pressed the other edge: after RL, small pass@k goes up, then the base model catches up as k grows. Under that reading, RL is a better sampler of traces the base already had, not a teacher of new behavior.
This University of Washington and Ai2 primer does not ship a new trainer. It treats post-training as three knobs (the base prior, reward granularity, and the prompt distribution) and turns them one at a time in tasks where the probability of a target string, or of a single AIME answer, can be measured exactly. The engineering questions are blunt. Where does sparse RL actually stall. Is the spurious-reward gain a general fact.
The RL optimizer is held fixed. There is no PPO-versus-GRPO bake-off.
The base prior is twisted with SFT. SFT+ raises the chance of the target (about 20% of quote-task samples are the target line, the other 80% are other movie lines so the model is not washed out). SFT- maximizes cross-entropy on the target until sampling probability is near 0%. The unmodified checkpoint is Base. The model list is Qwen2-1.5B/7B, Qwen3-1.7B/8B, Qwen2.5-7B-Instruct, OLMo 3, and Qwen2.5-7B-Math.
Two tasks. One is emitting a fixed movie quote, estimated over 10,000 samples. The other is AIME 2025 Problem 4: count ordered integer pairs in [-100, 100] for 12x² − xy − 6y² = 0; the answer is 117, scored over 128 samples. Sparse reward is binary. The quote task also adds a length penalty against rambling. Dense reward is Levenshtein distance on the quote, and a process reward model on the math item: Qwen2.5-32B-Instruct as judge, five gold-path milestones with weights 0.05 / 0.05 / 0.10 / 0.15 / 0.25, a 1.0 override for 117, 0.6 for the known near-miss 118, −0.3 for looping, −0.5 for a missing boxed answer. That dense signal is a ladder built for this one problem.
The spurious-reward setup replaces the verifier with Unif[0, 1]. Narrow D is 100 math prompts. Broad D is 10k prompts (WildChat for Qwen, the OLMo 3 RLVR mix for OLMo). OLMo is trained from Base, SFT, and DPO checkpoints to test whether lower starting entropy resists the scatter from random rewards.
On the quote task, sparse RL is a live demo of the coverage principle. Enough prior mass and the quote is learned. Exact zero mass and it is not.
| Model | Base prior | After sparse RL | SFT- |
| Qwen2-7B | 3.52% | 99.9% | 0% |
| Qwen3-1.7B | 0.48% | 10.0% | 0% |
| Qwen3-8B | 0.00% | 0.00% | 0% |
Qwen2-7B starts near 3.5%, sits on a roughly 40-step plateau, then hits 99.9%. Qwen3-1.7B starts near 0.5% and stalls at 10.0% under sparse rewards. Qwen3-8B starts at 0.00% and stays there under both sparse and Levenshtein rewards. SFT+ reaches 92%–100% across sizes.
Dense shaping helps when there is a foothold. Qwen3-1.7B Base reaches 48.8% exact match with edit distance, from a 0.48% prior. Qwen2-1.5B Base starts at 0.31% and only reaches 7.0% even with dense rewards; the small model still cannot climb out. SFT+ with the same dense reward underperforms sparse SFT+, because near-misses also score.
The AIME contrast is sharper.
| Setup | SFT+ | Base | SFT- |
| Before RL | 26.6% | 3.92% | 0.00% |
| Sparse reward | 85.9% | 10.2% | 0.00% |
| Process reward | 86.7% | 92.2% | 0.00% |
Sparse barely moves Base (3.92% to 10.2%). The process reward jumps it to 92.2%, a bit above SFT+ at about 86%. Generated traces include valid paths that are not copies of the SFT+ gold solution. SFT- never hits exact match (0% under both rewards), yet mean dense reward climbs to about 0.6: intermediate steps come back, the last hit does not.
Random rewards only look helpful under a narrow prompt set plus a base model that already leans the right way. Qwen math on 100 math prompts reproduces Shao's gains. The same random rewards on 10k WildChat prompts barely move MATH and hurt AMC. OLMo has no such math prior. On broad D, entropy spikes around step 400 and GSM8K, MMLU, and IFEval fall together. On narrow D, entropy dips slightly, GSM8K falls from 86% to about 32%, while MMLU (about 65 to 62) and IFEval (about 79 to 77) mostly hold. Narrow D sharpens whatever the base model already liked. That can raise scores or locally corrupt them. Neither setting creates new skills.
Three practical reads. Sparse verifiable rewards reweight mass the base model already puts on complete traces; they do not search regions the base almost never samples. Teaching that kind of behavior takes coverage first, or a dense reward the policy can climb. Shao's spurious-gain result is not a general law. It is tied to a narrow prompt set and a math-heavy Qwen prior. Random rewards on a broad prompt set mostly raise entropy and dump capability. The paper is a labeled diagram of the knobs, not a new trainer.
The sandbox is the point and the constraint. One frozen quote, one AIME item, and a process reward hand-built from that item's five-step gold solution, plus a 32B judge and an answer override. That density does not exist in ordinary multi-problem RLVR, so "dense rewards teach new behavior" is currently shown under highly informed shaping. When Qwen3-8B's prior on the quote is exactly 0%, even Levenshtein stays flat. Algorithm variants are out of scope, so the coverage threshold under GRPO is unknown. The caption of Figure 5 copies Qwen2-7B's 28.8%/3.5% onto a Qwen2.5-7B-Instruct figure. Table 1 is the source to trust.