Paper: RL for LLM Reasoning is Sparse Policy Selection, Not Capability Learning
mark_k · x · 2026-08-16
A new paper analyzes reinforcement learning (RL) for LLM reasoning and finds that it does not teach new strategies; instead, it merely redistributes probability over solutions the base model already contains. The beneficial edits are sparse (1–3% of tokens), concentrated at high-entropy decision points, and almost always promote tokens already in the base model’s top-5.
Building on this, the authors introduce ReasonMaxxer, an RL-free method that applies contrastive loss only at those entropy-gated spots. It requires only a few hundred rollouts, tens of problems, and minutes on a single GPU. Despite costing 1000× less, ReasonMaxxer matches or beats full RL across models and math benchmarks, reframing the problem as sparse policy selection rather than capability acquisition.
More from Research
- MIT CSAIL shares an overview of neural network fundamentals — MIT_CSAIL · 2026-08-17
- Interactive Diagram: Self-Attention vs. Cross-Attention Visualized — ProfTomYeh · 2026-08-16
- IR Papers Weekly Vol.169: Netflix builds LLM-native ranker, Yandex replaces 15+ models with one generative recommender — _reachsumit · 2026-08-16
- Why RL works for LLMs: Sparse but precise signals vs. noisy pre-training — burny_tech · 2026-08-16
- Battle Agents Platform Forces AI Agents to Fail — lannisterprince · 2026-08-16
- Developer open sources NoiseCheck, exposing statistical fallacies in model evals — Formal-King3851 · 2026-08-16