CorrGRPO: Correlation-Normalized GRPO for Multi-Reward Learning
Wenbin Hu, Huihao Jing, Haochen Shi, Yuxuan Liu, Haoran Li, Yangqiu Song
cs.LG, cs.CL
2026-09-29
CorrGRPO swaps covariances in GRPO's multi-reward denominator for correlations, curbing large-reward dominance; gains across code, tool calling, agent security, +4.21 avg Pass@1 at 7B.
People training reasoning models with GRPO spend their attention on reward design and weight tuning. Almost nobody looks at the denominator. When GRPO sums multiple reward components and normalizes by the within-group standard deviation, the variance of that sum decomposes into the sum of all pairwise covariances, so the denominator is quietly aggregating the full reward covariance matrix. The implicit mechanism is defensible: strongly correlated rewards carry redundant information, the denominator grows, and advantages shrink; negative correlations do the reverse. GRPO has been doing this adaptive scaling all along; nobody had written it down.
The failure mode is scale. A covariance is two standard deviations times a correlation, so a large-variance reward inflates the denominator through its own variance terms even when it is barely correlated with anything else. In the paper's three-reward example, one large-scale, weakly correlated reward (ρ≈0.11) accounts for 91.1% of the covariance sum, drowning out a strongly correlated pair (ρ≈0.98). The normalization claims to measure redundancy among rewards; in practice the component with the widest scale controls it.
CorrGRPO changes one thing: every covariance term in GRPO's denominator becomes the corresponding Pearson correlation coefficient. The centered total reward stays in the numerator, so the relative weights prescribed by the training objective are preserved.
In the Figure 2 example, the strongly correlated pair contributes about 8.89 times as much to the denominator as each weak pair, restoring the ordering that GRPO's covariance sum had inverted by scale. The dynamics also track training: as runtime success and all-tests-pass become increasingly correlated, the denominator grows and dampens redundant updates. The appendix derives this analytically for the two-reward case.
In theory, within any fixed rollout group, CorrGRPO's gradient equals GRPO's up to a shared positive constant; even the PPO clipping branches stay the same. The method's entire effect is a redistribution of gradient magnitude across groups. That is also why the denominator drops into DAPO, CISPO, and GDPO without touching their objectives.
Code generation, trained on LeetCodeDataset (2,641 problems), Qwen2.5-Coder-Instruct at four sizes, against GRPO and GDPO:
| Setting | Metric | GRPO | CorrGRPO |
| 7B | Avg Pass@1 over 4 benchmarks | 47.28 | 51.49 |
| 7B | LeetCodeDataset Pass@1 | 15.79 | 24.12 |
| 3B | Avg Pass@1 over 4 benchmarks | 43.29 | 45.56 |
| 0.5B | Efficiency (% faster than reference) | 60.71 | 67.86 |
Average Pass@1 over GRPO rises 2.09/0.80/2.27/4.21 points from 0.5B to 7B. At 7B, LeetCodeDataset Pass@1 goes from 15.79 to 24.12 against a 14.91 base. CorrGRPO holds every nondominated point on the correctness-efficiency Pareto frontier.
Tool calling, trained on RLLA-4K with zero-training transfer to API-Bank. RLLA-4K all-exact: 63.38→67.61 (Qwen2.5-7B), 57.75→59.15 (Qwen3-4B-Thinking), 61.97→63.38 (Qwen3-8B). API-Bank averages gain 1.14/3.94/2.26 points; the largest single gain is Qwen3-4B on API-Bank v3, 52.24→64.08.
Agent security, trained in AgentDojo with equal-weight utility and security rewards, transferred to ASB and InjecAgent:
| Metric | GRPO | CorrGRPO |
| AgentDojo 3B Joint Accuracy | 60.13 | 89.62 |
| ASB 7B Joint Accuracy | 31.38 | 47.88 |
| InjecAgent 7B overall ASR (lower is better) | 13.53 | 6.17 |
On AgentDojo at 3B, clean utility climbs from 52.38 to 95.24 while ASR drops to 1.54, capturing utility and security at once. The appendix also shows slower policy-entropy decay under CorrGRPO, most visible on AgentDojo and LeetCodeDataset, consistent with the known link between entropy collapse and stalled exploration.
The change is roughly a dozen lines in the denominator (Appendix B has the diff): no new hyperparameters, no changes to clipping or KL. Any pipeline already running multi-reward GRPO or DAPO can A/B it directly. The conceptual contribution is real too. Reward-scale alignment is an old pain in multi-objective training, usually handled by manually rescaling components or reweighting on the fly; this paper shows GRPO's group normalization is already aggregating covariances, and that is where the scale imbalance lives and where the fix belongs. It is a clean, well-scoped improvement to a neglected statistical detail, not a paradigm shift.
Author-stated: gains on security benchmarks do not establish safety in real deployments or against unseen attacks, and correlation normalization cannot rescue biased or misspecified rewards.
Further concerns from a close read. Some gains are small: +0.80 average Pass@1 for 1.5B code, +1.25 ASB Joint Accuracy for Qwen3-8B, with no variance or significance reporting across runs. A few cells lose to GRPO: 1.5B LiveCodeBench v6 is 10.29 versus 13.14, and the 3B and 7B executable rates are slightly lower. Combining with GDPO drops average Pass@1 from 48.88 to 48.21; the paper concedes the integrations establish compatibility rather than guaranteed gains. The seven coding reward weights (0.60 pass, 0.30 AST, 0.20 efficiency, and so on) are hand-set with no ablation. Most notably, MO-GRPO, named in related work as the closest variance-reweighting approach, never appears in the experiments.