Dense process rewards lose to standard GRPO at matched compute, 1,668-run study finds
CWRU · hf · 2026-11-11
Using a matched-compute protocol (equal generated tokens and GPU-hours) across 1,668 preregistered runs on six models, the study finds dense process rewards like Monte-Carlo rewards underperform standard GRPO: continuations alone consume 91% of generated tokens. On Qwen2.5-1.5B, MC rewards lead per update but end 14.3 points lower at equal tokens and 7.8 lower at equal time. A per-token gradient test ranks methods correctly but overestimates each by 1-2 points. Instead, training on prompts with predicted success near 1/2 yields consistent gains (+1.7 on Qwen3-0.6B, +0.8 on Qwen2.5-1.5B).
More from Research
- Wang 2026 paper: monolithic active-cavity laser-as-a-sensor oxygen detection with subnanoliter volume — jwt0625 · 2026-11-11
- Laser-as-a-sensor: monolithic VCSEL oxygen detector shrinks sampling volume to subnanoliter — jwt0625 · 2026-11-11
- AI×Antibodies hackathon in Oslo offers JapanFold, Anthropic credits, $20K lab validation — DavidBennett__ · 2026-11-11
- Which benchmarks do humans still win against frontier AI? Reddit debates AGI criteria — ImproveOurWorld · 2026-11-11
- EarthRecord benchmark tests 12 multimodal agents on four decades of raw satellite imagery — AIEyesTech · 2026-11-11
- WaveGen: codec-free end-to-end diffusion hits SOTA text-to-speech via Spectral Forcing — Ajou · 2026-11-11