Study: RL Doesn't Expand LLM Reasoning Boundaries, It Reweights Existing Strategies
burny_tech · x · 2026-10-28
A new arXiv paper (Ziheng Cheng, Somayeh Sojoudi et al.) resolves conflicting evidence on whether RL expands LLM reasoning or merely reweights existing capability.
Key findings:
- Across Qwen and Gemma families, math RL training yields cross-domain gains yet Pass@1 improvements can coincide with lower Pass@N than the base model; coverage at large sampling budgets rises on some tasks and falls on others.
- Trace analysis shows the shift comes from which reasoning strategies the model selects, not new strategy emergence.
Theory:
- A two-stage autoregressive policy model separates strategy selection from execution, proving RL's implicit bias reshapes strategy preferences—boosting some tasks while suppressing strategies others need.
- This mechanism explains transfer, forgetting, and coverage shifts without expanding strategy support.
- The authors derive log-sigmoid and log-linear scaling laws for RL compute and validate their predictive power.
Related event: Berkeley Paper: RL Reweights Rather Than Expands LLM Reasoning Boundary(3 posts)→
More from Models
- Grok 4.7 Sweeps Top 3 on VulcanBench Frontier v4, Beating Opus 5.5 and Sonnet 5.5 — XFreeze · 2026-10-28
- Rumors: OpenAI and Anthropic internal models gain reasoning via RL in games and simulated environments — imjustnewatai · 2026-10-28
- Users notice ChatGPT almost always ends replies with "One question:" — Angaisb_ · 2026-10-28
- Community wishlist for next Gemma: better multilingual, OCR, fewer refusals — MaziyarPanahi · 2026-10-28
- Mystery model Space Bunny revealed as early MiniMax M3.1 Flash, free on EvoX Agent Oct 28–30 — lxfater · 2026-10-28
- Accusations fly that Artificial Analysis rankings systematically underrate open-weight models — aiamblichus · 2026-10-28