Recycling Reward-Saturated Data Boosts GRPO by Up to 9% on Qwen3 Models
shangbinfeng · x · 2026-10-22
Researchers from the University of Washington show that reward-saturated RL data — where all sampled responses earn equally high rewards and GRPO's group-relative signal vanishes — can be recycled rather than discarded.
Key finding: Intervening at rollout generation works best: nudging the policy to produce high-quality but incorrect solutions injects low-reward rollouts into saturated groups as negative samples, improving GRPO by 6.4%-9.0% across Qwen3-1.7B and 4B. Interventions across four pipeline levels (data, rollout, reward, advantage) were systematically tested; raising temperature or adding auxiliary rewards restores non-zero advantages but less consistently.
Additional results: mixing saturated with unsaturated data yields +1.8-3.1 pts, the method transfers to Llama-3.1-8B, and it enables a self-evolving loop — newly solved prompts (443 by step 80) join the saturated pool while AIME25 climbs from 22.1 to 32.7.
Paper and code are available on arXiv and GitHub.
More from Research
- EMNLP 2026 paper uses prediction-powered inference to quantify how much human annotation LLM judges save — shi_weiyan · 2026-10-22
- Ask-to-Distill: synthesizing questions for answer-free on-policy distillation — UIUC-CS · 2026-10-22
- RFD4 team swaps self-conditioning for recurrent updates, echoing looped flows — AllThingsApx · 2026-10-22
- MIT ML technique breaks records for 2-6 week weather forecasts by learning to correct model errors — LesterMackey · 2026-10-22
- Harvard undergrad's ML subseasonal weather forecasting work, started at Microsoft Research, lands in Nature — LesterMackey · 2026-10-22
- Chatlet: generative UI that emerges from group chats to power collaborative decisions — HaijunXia · 2026-10-22