On-Policy Parameter Update Direction Underlies Generalization in LLM Post-Training
Shufan Shen, Zhongni Hou, Junshu Sun, Yufei Zhang, Wei Lin, Guojun Yin, Qingming Huang, Shuhui Wang
cs.LG
2026-09-29
50 GRPO steps record each weight's sign; SFT then moves only that way. On Qwen3-8B, four math contests average 41.67 in 8.9h, above 100-step GRPO at 40.31 (19.3h).
On-policy post-training updates a model on answers it has just sampled. GRPO, which uses within-group relative advantages as a policy gradient, and OPD both belong to this family. They usually generalize further on reasoning tasks than SFT. SFT fits a frozen set of target answers. Training is cheap, and the targets can be trajectories written by a stronger model, but scores often drop on questions that set did not cover. A line that has stuck is that SFT memorizes and RL generalizes.
Earlier accounts either isolate one loss term, such as the reverse KL in OPD or the negative samples in GRPO, or they describe which weights move and how little the spectrum shifts. The description stays inside on-policy training. Too many moving parts make a piece-by-piece comparison expensive and fragmented. This paper asks one narrower question: can that generalization gap be stored as the cumulative sign of each parameter, then handed back to SFT?
The gradients point differently. Let s be the gradient of the trajectory log-probability. The on-policy policy gradient is the covariance between the advantage A and s. Advantage here is how much better a sampled answer is than the current policy's average. The policy keeps changing, so the advantage distribution changes with it, and the update direction keeps turning. SFT draws trajectories from a fixed teacher, so its gradient stays roughly aligned with s.
Cosines on Qwen3-1.7B trained on DeepMath match that split. Across stages, SFT updates are positively correlated, while OPD and GRPO are nearly orthogonal. Summed over training, SFT's direction has cosine near 1.0 with itself. On-policy cumulative updates sit near 0.5. The on-policy run keeps revising the cumulative direction.
OPSFT uses that cumulative sign as a mask. v is the element-wise sign of parameters after on-policy training minus parameters before it, so each weight is marked -1, 0, or +1. SFT still computes a gradient g on the teacher trajectories. A descent step moves along -g, and OPSFT keeps only components whose update sign agrees with v. The rest are zeroed. Regularization inside the optimizer can bend a step that the gradient mask already allowed, so the sign is enforced again after every optimizer update. Weights with sign 0 stay put.
A mask that shares only the update locations, and ignores the sign, does worse than plain SFT. Knowing which weights move is not enough. The sign is.
The experiments use the mask in two places. For speed, 50 GRPO steps fix v, then 100 constrained SFT steps run with no further sampling. For a model that has already finished GRPO, new trajectories are applied along the original v, so ordinary SFT does not wash out the on-policy skill. The SFT trajectories come from Qwen3-30B-A3B.
In Figure 3, direction-constrained OPSFT lands well above vanilla SFT and close to the GRPO run that supplied the direction. The efficiency setup freezes that direction early. On DeepMath, GRPO runs 100 steps, SFT and DFT (an improved SFT baseline) run 700, and OPSFT is 50 GRPO steps plus 100 constrained steps. Reported time includes finding the direction.
| Model | Method | Time | Mean of four |
| Qwen3-1.7B | GRPO / SFT / DFT / OPSFT | 4.9h / 3.5h / 3.9h / 2.3h | 12.09 / 13.15 / 13.16 / 15.11 |
| Qwen3-4B | same order | 16.5h / 8.1h / 8.6h / 6.4h | 38.96 / 34.22 / 34.69 / 40.11 |
| Qwen3-8B | same order | 19.3h / 13.2h / 13.9h / 8.9h | 40.31 / 38.41 / 38.97 / 41.67 |
| Distill-Llama-8B | same order | 18.2h / 13.6h / 14.1h / 8.1h | 25.21 / 18.02 / 19.90 / 26.56 |
The four benchmarks are AIME24, AIME25, HMMT25-Feb, and HMMT25-Nov. On Qwen3-8B, OPSFT reaches 41.67 in 8.9 hours. GRPO reaches 40.31 in 19.3 hours. The mean hides losses: AIME25 is 42.92 for OPSFT, below GRPO at 44.58 and SFT at 44.48, and HMMT25-Feb is 26.67, below GRPO at 28.33. AIME24 at 57.92 and HMMT25-Nov at 39.17 carry the average. At 1.7B, plain SFT already beats this GRPO run, 13.15 against 12.09.
After 100 GRPO steps, another 700 steps on DeepMath trajectories pull the larger models down if the update is unconstrained. Qwen3-4B falls from 38.96 to 36.25 under SFT and rises to 41.36 under OPSFT. Qwen3-8B goes from 40.31 to 37.50 under SFT and to 42.39 under OPSFT. The DeepSeek distill drops from 25.21 to 16.36 under SFT; OPSFT finishes at 26.15. The 1.7B model moves the other way: extra SFT takes it from 12.09 to 13.54, and OPSFT reaches 16.77.
On code, Qwen3-4B trained on Eurus is scored with HumanEval+, MBPP+, and LCBv6. From scratch, the means are GRPO 57.19 (20.3h), SFT 54.30 (9.8h), and OPSFT 58.07 (9.5h). Continued training on a GRPO checkpoint sends SFT to 48.71 and OPSFT to 60.05. MBPP+ under that OPSFT run is 68.20, still under the 69.04 it started from.
Out of domain is a different story. The same DeepMath models are tested on IFEval, ARC, HaluEval, Hellaswag, Winogrande, and PIQA. Qwen3-4B means are base 73.43, SFT 73.65, GRPO 73.95, OPSFT 74.11. Qwen3-8B means are 75.85, 75.54, 76.20, 76.39. OPSFT leads GRPO by less than 0.2 points. The extra math reasoning barely appears on these tasks.
Reuse has a sharp boundary. Overlap of update locations is about 0.4 both for DAPO versus DeepMath and for the code set Eurus. Direction cosine inside math is only about 0.2. OPSFT trained on DeepMath with the DAPO direction is slightly stronger than OPSFT that uses DeepMath's own direction. Swap in the Eurus direction and math gets clearly worse. Same-domain masks transfer. Cross-domain masks do not. Later GRPO snapshots make better masks, but the gain from step 50 to step 100 is larger than the gain from step 100 to step 150.
Precision makes the sparsity concrete. On Qwen3-8B in BF16, SFT updates 2.702% of parameters for a mean of 38.33. OPSFT updates 0.408% for a mean of 40.94, 24.59 points above the base mean of 16.35. In FP32, SFT updates about 89.3% of parameters (mean 38.41). OPSFT updates 9.454% (mean 42.81), a bit above the GRPO run in the same table at 42.01, which updates 9.499%. The sign mask pulls SFT's footprint into the same range as GRPO.
Response length is similar for SFT, OPSFT, and GRPO. OPSFT moves closer to GRPO on accuracy variance across samples, and on how often case-splitting and logical connectives show up. The chain of thought does not get longer.
Sampling is the expensive part. High-quality trajectories are often already available. A short GRPO run fixes the direction once, and the rest of training does not sample. A finished on-policy model can take new trajectories without a full SFT-then-RL restart from the base weights. On the same trajectories, OPSFT posts a higher math mean than DFT, which trains for many more steps, and it finishes sooner.
The GRPO objective is unchanged. Once the cumulative signs are known, SFT restricted to that line can match a short GRPO run on math and code means. Sharing locations alone cannot.
The main text parks the limitation write-up in Appendix C.2 and names only two follow-ups: which pieces of an on-policy method find this direction, and how properties of the trajectories affect generalization once they pass through OPSFT.
The direction is not free. It still costs an on-policy run, and an earlier snapshot is a weaker mask. GRPO in the efficiency table is only 100 steps, while the second stage of OPSFT trains on trajectories written by Qwen3-30B-A3B. Against SFT and DFT on those same trajectories, the sign constraint is a clean gain. Against GRPO, teacher quality and the sign mask are tangled, so beating GRPO is not a pure direction result.
The out-of-domain lift is under 0.2 points. Transfer onto those six tasks is essentially flat. Two math datasets agree on direction only at cosine about 0.2, so there is more than one useful direction, and the word underlies in the title is stronger than the intervention shows. On-policy training itself keeps a cumulative cosine near 0.5. It is still turning, while OPSFT freezes one snapshot. AIME has thirty questions, and the tables do not report error bars. A one- or two-point swing on a single contest can be noise. At 1.7B, SFT already beats this GRPO setup, so the slogan that SFT memorizes and RL generalizes does not hold without a scale attached.