RL Shows a Matthew Effect on LLMs: NGU Adaptive Sampling Allocates Compute to Hard Problems
vwxyzjn · x · 2026-09-16
A new arXiv paper by Noukhovitch, Ivison, Lambert, and Courville shows RL training gains on LLMs are highly uneven: large improvements on problems the model already solves, minimal gains on hard ones — dubbed the "Matthew Effect" in RL for LLMs.
The authors argue modern RL wastes compute on easy problems and should dynamically reallocate sampling. They propose Never Give Up (NGU), an adaptive sampling method that keeps generating samples for a problem until one is correct; with asynchronous RL this naturally filters easy problems and funnels compute toward harder ones, with best practices around off-policy robustness.
On Deepscaler, NGU improves performance per compute, especially on hard problems. On the coding benchmark Manufactoria, standard GRPO with per-test rewards fails to fully solve mixed-difficulty tasks, while NGU iteratively masters harder tests until it solves problems completely.
Related event: Research Reveals RL Matthew Effect; NGU Sampling Targets Hard Problems(4 posts)→
More from Research
- Researcher: Hacked model behavior stems from RL training, not loyalty — dhadfieldmenell · 2026-09-16
- Google's 145-page doc details how researchers use Gemini for scientific discovery — dzscholar · 2026-09-16
- AI in Science report: bottlenecks shift downstream as hypothesis backlogs pile up — soumitrashukla9 · 2026-09-16
- Study: API-based audits of ChatGPT, Claude and Gemini don't transfer to chatbot UIs — kenziyuliu · 2026-09-16
- Researcher ports dREG peak-calling tool to GPU-backed pydreg using Claude and Codex, note on bioRxiv — anshulkundaje · 2026-09-16
- Astera Institute's third residency cohort offers up to $2M per researcher — juanbenet · 2026-09-16