RL Shows a Matthew Effect on LLMs: NGU Adaptive Sampling Allocates Compute to Hard Problems
vwxyzjn · x · 2026-09-16
A new arXiv paper by Noukhovitch, Ivison, Lambert, and Courville shows RL training gains on LLMs are highly uneven: large improvements on problems the model already solves, minimal gains on hard ones — dubbed the "Matthew Effect" in RL for LLMs.
The authors argue modern RL wastes compute on easy problems and should dynamically reallocate sampling. They propose Never Give Up (NGU), an adaptive sampling method that keeps generating samples for a problem until one is correct; with asynchronous RL this naturally filters easy problems and funnels compute toward harder ones, with best practices around off-policy robustness.
On Deepscaler, NGU improves performance per compute, especially on hard problems. On the coding benchmark Manufactoria, standard GRPO with per-test rewards fails to fully solve mixed-difficulty tasks, while NGU iteratively masters harder tests until it solves problems completely.
Related event: Research Reveals RL Matthew Effect; NGU Sampling Targets Hard Problems(4 posts)→
More from Research
- ICML paper: finite context windows impose impossibility results for adversarial regret — burny_tech · 2026-11-02
- WASABI: Blei lab trains diffusion models to sample Boltzmann distributions with 3.3x fewer energy evaluations — skoularidou · 2026-11-02
- A complete math roadmap for machine learning: linear algebra, calculus, probability — TivadarDanka · 2026-11-02
- A base LLM plus 11 lines of code is all you need for a decision model — helloiamleonie · 2026-11-02
- Editor fully resigns from JMLR, arguing AI reviews beat rushed humans and journals need a rethink — roydanroy · 2026-11-02
- flowl: a probabilistic programming language built on optimal transport, not distributions — burny_tech · 2026-11-02