GUPO: Gradient Uncertainty-aware Policy Optimization for Post-Training Large Language Models
Peizheng Guo, Jianqi Zhang, Xingyu Zhang, Yun Fan, Jiahuan Zhou, Changwen Zheng, Wenwen Qiang
cs.LG
2026-08-18
GUPO models each GRPO group gradient as a random variable and downweights uncertain groups. On three 1.5B/7B math models it lifts six-benchmark Pass@1 by 1.8-3.0 over GRPO.
GRPO is the default post-training recipe for reasoning models: sample a group of answers per query, weight them by group-relative advantage, then average the resulting group gradients across a mini-batch. A team at the Institute of Software, CAS, UCAS, and Peking University's Wangxuan Institute finds that those group gradients often point in opposite directions.
They compute pairwise cosine similarity on group gradients. Of 563 pairs, 262 are negative, 46.5%. High-conflict mini-batches show a lower median validation ΔNLL and some negative updates. The question becomes: when groups disagree, how do you form a trustworthy aggregated step?
Vanilla GRPO treats each group gradient as a deterministic vector and averages them equally. GUPO estimates how uncertain each group gradient is, then reweights by that reliability.
A full-model posterior is intractable. GUPO puts a Bayesian approximation only on Φ, the last trainable layer. A diagonal empirical Fisher plus δI stands in for the Hessian, giving a diagonal Gaussian around the current parameters. Monte Carlo draws of Φ, with other layers frozen, recompute each query's group gradient and yield a mean plus per-dimension variance.
Precision is the inverse variance, raised to power s to get evidence. Following subjective logic and evidential deep learning, per-dimension belief is evidence over Dirichlet strength, and group uncertainty is ub = K / Sb, with K the gradient dimension. Low-uncertainty groups get more weight. The final coefficient mixes the original 1/B average with that uncertainty weight through η=0.1, a light correction rather than a replacement of GRPO.
A sweep picks s=0.5 and η=0.1. Training uses learning rate 1e-6, weight decay 0.01, global batch 256, on H100 GPUs. Bases: DeepScaleR-1.5B-Preview, DeepSeek-R1-Distill-Qwen-1.5B, DeepSeek-R1-Distill-Qwen-7B. Baselines include GRPO, Length Penalty, ReST-MCTS, GVPO, Dr.GRPO, GCPO, and MRT.
Pass@1 averaged over six math benchmarks, versus the untrained Vanilla:
| Base | Vanilla | GRPO | next-best GCPO | GUPO |
| DeepScaleR-1.5B | 60.3 | 60.7 | 62.3 | 63.4 (+3.1) |
| R1-Distill-Qwen-1.5B | 51.3 | 52.4 | 53.7 | 55.4 (+4.1) |
| R1-Distill-Qwen-7B | 68.6 | 69.6 | 70.9 | 71.4 (+2.8) |
Contest sets move more. DeepScaleR AIME 2024: Vanilla 42.8, GRPO 44.5, GUPO 48.7. The 1.5B Distill MATH500: Vanilla 80.1, GRPO 80.3, GUPO 84.9. 7B AIME 2025: Vanilla 50.2, GRPO 51.7, GUPO 53.9. Length Penalty drops on all three bases. Figure 5 shows GUPO still beating GRPO on high-conflict mini-batches.
For teams already running GRPO, this is a drop-in aggregation fix: same sampling, same advantages, different mixing of per-query group gradients. The mean lift over GRPO is 1.8 to 3.0 points, and 0.5 to 1.7 over GCPO, the strongest published control in the table.
The cost is a last-layer Fisher diagonal plus M extra backward passes per step. Wall-clock is not reported. η=0.1 means most of the update is still the original equal average; uncertainty is a mild bias.
There is no dedicated limitations section. The posterior lives only on the last layer, and a diagonal Fisher drops parameter correlations; group-gradient covariance is flattened to per-dimension variance. Conflict versus ΔNLL is an association, not a causal proof. Evaluation is verifiable math only: no code, tools, or open-ended tasks. All three bases are small. On 7B the Vanilla gap is only 2.8 points, and GRPO itself beats Vanilla by 0.4 on DeepScaleR, so a weak baseline can inflate relative gains. The paper never states M in the main text. The η=0.1 optimum also says that fully replacing the equal average with uncertainty weights is not necessarily better.