When RL data saturates: systematic study of extracting learning signals at data, rollout, reward and advantage levels

shangbinfeng · x · 2026-10-22

As models get stronger, RL training data saturates: all rollouts correct, zero reward variance, vanishing learning signal. This work systematically studies interventions across four levels — data (perturb/rephrase prompts), rollout (generate wrong answers), reward (RM or auxiliary rewards), and advantage (zero padding) — to keep extracting useful signals from saturated data.

Original post →

When RL data saturates: systematic study of extracting learning signals at data, rollout, reward and advantage levels

More from Research