Dense rewards lose to standard GRPO at matched compute, arXiv paper finds
burny_tech · x · 2026-11-12
A pre-registered study compares dense process rewards against standard GRPO under a compute-matched protocol (equal generated tokens and GPU-hours; reward rollouts cost 91% more tokens). Once compute is equalized, rankings flip: the Monte-Carlo reward, which looks better per update on Qwen2.5-1.5B, ends up 14.3 points lower at equal tokens and 7.8 points lower at equal training time, and none of four dense-reward variants beats plain GRPO with as much training time as two points of overestimated per-update gains. Instead, training on prompts with predicted success near 1/2 reliably improves accuracy (+1.7 on Qwen3-0.6B, +0.8 on Qwen2.5-1.5B), replicating in five of six pre-registered tests. For RLVR practitioners, sparse GRPO updates and difficulty-based prompt filtering beat dense feedback rollouts.
Related event: Dense Rewards Fail to Beat Standard GRPO Under Equal Compute(2 posts)→
More from Models
- Hassabis RTs tease that Gemini 4 carbon is 'awesome', results coming soon — demishassabis · 2026-11-12
- Google exec introduces Gemini 4 Carbon, its strongest frontier model for complex reasoning — goyalshaliniuk · 2026-11-12
- Aside's Computer Use agent claims SOTA on Scale AI's Cua Driver Bench — brandon_galang · 2026-11-12
- Inflect v3 ships six-voice local TTS in just 34M parameters, Apache 2.0 licensed — b111ue · 2026-11-12
- Qwen 3.8 27B on a laptop can read model weights, replicating the Weight-Space Programming paper — dl_weekly · 2026-11-12
- KOL predicts Gemini will be the top frontier LLM by year-end as rivals stumble — iruletheworldmo · 2026-11-12