UW Research Recycles Reward-Saturated Data for RL, Boosting Qwen3 by Up to 9 Points

A University of Washington team (Ziyuan Yang, Yike Wang, Shangbin Feng, Yulia Tsvetkov) released the paper "Save Your Saturated Data: Learning Beyond Reward Saturation," systematically studying an increasingly pressing problem: as models get stronger, RL training data keeps saturating—all rollouts answer correctly, rewards have zero variance, advantages approach zero, and the learning signal dries up. The work experiments across four levels—data, rollout, reward, and advantage—to extract further learning signal from saturated data.

Confirmed

• Most effective approach: run RL only on saturated MATH problems where every rollout is correct, having the policy model itself write a wrong solution to serve as a negative sample within the group—beating GRPO by 9.0% on Qwen3-1.7B and 6.4% on 4B (across 8 math, reasoning, and metrics benchmarks).
• Ablation analysis: non-zero advantages alone are not enough—padding with fake 0-reward samples raises advantage variance but underperforms GRPO; replacing only the final answer is harmful; raising temperature or rewriting problems gives unstable results. The takeaway: useful negative samples need coherent, plausible, complete failure trajectories.
• Mixed training: the negative-sample method still works when training on a mix of saturated and unsaturated data, adding another 1.8–3.1 points, lifting AIME25 from 22.1 to 32.7, and transferring to Llama-3.1-8B.
• Self-evolution loop: newly solved problems are added to the saturated pool, contributing 443 new problems by step 80.

Why it matters

Data saturation is one of the core bottlenecks of RL scaling. This study offers an actionable way to recycle saturated data that originally carries no learning signal, and it is compatible with mixed training, cross-model transfer, and self-evolution loops—directly useful for improving RL training efficiency as models get stronger.

2026-10-22 ~ 2026-10-22 · 5 related posts

• When RL data saturates: systematic study of extracting learning signals at data, rollout, reward and advantage levels — shangbinfeng · 2026-10-22
• [source] Let the policy write wrong solutions: negative rollouts beat GRPO by 9.0% on Qwen3-1.7B using only saturated MATH data — shangbinfeng · 2026-10-22
• [source] Recycling Reward-Saturated Data Boosts GRPO by Up to 9% on Qwen3 Models — shangbinfeng · 2026-10-22
• RL ablations: zero-padding fake negatives underperforms GRPO; good negatives need coherent failed trajectories — shangbinfeng · 2026-10-22
• [source] Self-evolving RL loop on saturated data lifts AIME25 from 22.1 to 32.7 and transfers to Llama-3.1-8B — shangbinfeng · 2026-10-22