Make Sparse Rewards Count: Density-Aware Reward Aggregation for Multi-Reward RL
Tong Zheng, Skylar Zhai, Zhan Cheng, TianMing Sha, Youling Huang, Shuo Zhou, Shaotong Qi, Jingcheng Liang, Xuwei Ding, Pengcheng Xu
cs.LG, cs.CL
2026-10-01
DARA weights sparse rewards by inverse-square-root active-group density, equalizing advantage energy; format and length compliance arrive up to 65% faster than GDPO.
Post-training RL for LLMs increasingly juggles several behavioral objectives at once: answer correctness, output format, response length, reliable tool use. GRPO-family methods estimate advantages from within-group comparisons over sampled responses and skip the value model. With multiple rewards, the aggregation order matters. GRPO sums the rewards first and normalizes the scalar, which maps distinct reward combinations to identical advantages; GDPO normalizes each reward within its group first and then sums, preserving per-dimension relative information.
Yet GDPO's own experiments show uneven progress across objectives: tool-calling accuracy arrives well before format compliance, and GDPO attributes this to the model favoring easier objectives, which modest manual weight tuning does not reliably fix. This paper pushes the question to the batch level: after reward-wise normalization, what imbalance remains, and can weights be derived from the mechanism instead of tuned by hand?
The key quantity is active-group density: the fraction of rollout groups in which a reward takes non-identical values, since an all-success or all-failure group yields zero relative advantage after normalization. Under idealized GDPO normalization, the derivation shows every active group contributes exactly G−1 units of advantage energy (the sum of a reward's squared advantages over the batch), so the batch total equals groups × density × (G−1), strictly proportional to density. Reward-wise normalization equalizes the per-group contribution; the batch-level total still shrinks for rewards that rarely fire.
Why does density vary so much? For a binary reward with per-rollout success probability p, a group is informative with probability 1−p^G−(1−p)^G, so both very hard and nearly saturated rewards have low density while mid-difficulty ones peak. This lines up with the training dynamics: the format reward improves fastest while its density peaks, and the signal thins as it saturates.
DARA matches each reward's energy to the densest reward in the batch, which solves to wk = min(wmax, sqrt(πref/πk)), with πref the maximum density. An infrequently active reward gets amplified by the inverse square root of its density whenever it does produce a useful comparison. Weights are recomputed per rollout batch and track reward activity through training; the policy objective itself is untouched. Two variants: DARA-Sym multiplies both positive and negative advantages by wk and is the direct energy-matching implementation; DARA-Asym, the main method, amplifies only positive advantages and keeps negative ones at GDPO scale. The motivation is conflict handling: a correct but overlong response picks up a negative length advantage, and symmetric scaling would strengthen that negative pressure along with the positive ones. wmax is set to 5.
Tool calling (ToolRL's 3,920 training examples, Qwen2.5-1.5B/3B-Instruct, binary format reward plus correctness in [-3,3], evaluated on BFCL-v4):
| Metric | GDPO | DARA |
| Steps for format reward to hit 0.8 (1.5B) | 19 | 14–15 |
| Avg format at step 60 (1.5B) | 90.11% (best baseline) | 96.02–96.06% |
| Avg format at step 100 (1.5B) | 97.11% | 97.55–97.90% |
| Multi-turn format at step 100 (1.5B) | 91.40% | 92.77–94.17% |
GRPO needs 34 steps for the same threshold. At step 60, DARA leads average accuracy at 50.69–50.94% against a best baseline of 50.50%. On the 3B model, DARA-Sym posts the best average format at 97.22% with accuracy 54.15%, essentially tied with DVAO's best 54.22%. Adding a third reward (think block capped at 16 words) drops GDPO's average format from 97.11% to 94.39%; DARA-Asym only slips from 97.90% to 96.93%, so the new objective does not crowd out the old ones.
Mathematical reasoning (DeepSeek-R1-1.5B, Qwen3-4B-Instruct, and DeepSeek-R1-7B on DeepScaleR data, two binary rewards, correctness and length within 4,000 tokens, macro-averaged over five benchmarks):
| Metric | GDPO | DARA |
| Steps to 99% length compliance (R1-7B) | 142 | 50–56 |
| Steps to 95% held-out compliance (R1-7B) | 70 | 50 |
| Exceed rate at step 50 (R1-7B, DARA-Sym) | 6.82% | 1.09% |
| Joint score at step 50 (R1-7B, DARA-Asym) | 56.48% | 57.83% |
The joint score requires answers that are both correct and within the length limit. DARA-Asym takes the top accuracy and joint score on both R1 models (58.31% accuracy on R1-7B vs GDPO's 57.45%). The fixed-weight ablation is the decisive control: pinning the length weight at 5 does lower the Exceed rate (3.14% vs DARA-Asym's 4.21%) but drops accuracy from 48.03% to 46.84%. The speedup does not come from simply inflating a weight; the dynamic calibration buys faster convergence at a much smaller accuracy cost. The theory also predicts that at a fixed response budget, larger rollout groups raise density and accelerate learning, and the G ∈ {4,8,16,32} sweeps confirm it, with DARA ahead at every group size.
Multi-reward weighting has been a hand-tuned black box. DARA turns it into a formula computed from rollout statistics: derived, capped, recomputed every batch. Integration cost is low since only the reward-aggregation step changes while the rest of the verl stack stays aligned. Teams aligning sparse objectives such as format, length, or safety benefit most in the endgame, exactly when the target saturates and its signal thins; the method is orthogonal to policy-update tricks (GSPO, DAPO) and to task reward design, so it composes with them.
This is a convergence-speed improvement, not a jump in final capability: by step 100 most methods have converged and the gaps narrow.
The density-energy identity holds under idealized conditions (ε = τ = 0, sample standard deviation with divisor G−1). The main method, DARA-Asym, falls outside the exact energy-matching result and is only bounded above and below; the positives-only design rests on a qualitative argument. The cap wmax = 5 binds late in training (steps 48, 59, and 62 for the three models), after which the energy ratio drifts below 1 to a median of 0.39–0.49, precisely in the strict-compliance regime where DARA and GDPO separate most, and wmax itself becomes a new hyperparameter to tune. Gains concentrate in speed: at step 50 on Qwen3-4B-Instruct, GRPO leads accuracy (62.14% vs DARA-Asym's 60.30%), and the two variants trade accuracy against compliance, with Sym pressing the Exceed rate harder at lower accuracy and Asym the reverse. Experiments cover 1.5B–7B models, two task families, and mostly binary rewards; larger scales, more reward dimensions, and continuous rewards are untested. A crowd of concurrent methods (DVAO, SAW, SA-MRPO, GD²PO, among others) targets the same problem, and the head-to-head here covers only two reimplemented baselines.