UCLA's ReSPO fixes sign-dependent gradient starvation in off-policy RLVR

ucla-cmllab · hf · 2026-10-09

UCLA CMLL Lab proposed ReSPO (Reshaped Sequence Policy Optimization), addressing gradient starvation in off-policy RLVR.

RLVR frequently reuses rollouts across policy updates, and the authors show clipped policy optimization suffers sign-dependent gradient starvation: clipping suppresses under-generated positive responses in the low-importance-weight tail while letting severely over-generated negative responses dominate the high-weight tail.
ReSPO replaces clipping with a smooth two-branch sequence-level kernel derived from an α-divergence variational objective plus exponential variance-control tilt: the positive branch preserves nonzero gradient weight for under-generated positives, the negative branch suppresses over-generated negatives.
On dense and MoE Qwen3 models, ReSPO accelerates early optimization, improves final training scores, and achieves higher held-out benchmark performance under rollout reuse, learning effectively from long positive reasoning trajectories early in training.

More from Research