New theory of RL post-training shows REINFORCE coverage collapse and reference-reset fix
gaotianyu1350 · x · 2026-11-12
Researchers including Audrey Huang, Sham Kakade, Sadhika Malladi, Akshay Krishnamurthy, and Dylan Foster publish arXiv:2611.14467, a theory of policy optimization in LLM post-training.
Key results:
- Coverage-preserving guarantees: near-optimal learning with sample/gradient complexity governed by the base model's response coverage—formalizing post-training as amplifying capabilities accessible by sampling the base model.
- Coverage collapse: REINFORCE (the basis of PPO/GRPO) can drive the probability of optimal responses near zero even as reward improves; convergence can take exponentially many steps even with exact gradients and KL regularization.
- Reference resets: repeatedly resetting the KL reference to the latest policy mitigates collapse. The authors formalize this as Approximate Policy Mirror Descent (APMD), enabling stronger per-subproblem regularization while converging to global optimality.
The work turns the heuristics behind modern RL objectives into actionable algorithm-design principles and explains why base model quality drives post-training.
More from Research
- Berkeley open-sources RoboSimStudio: AI agents program industrial robots without training data — berkeley_ai · 2026-11-12
- LAION releases 1.48B-sample audio corpus; CLAP2-L hits 53.6% on MAEB-30 — serrjoa · 2026-11-12
- Allen-Zhu's new paper: Canon layers boost reasoning 2x, even random convolution weights help — teortaxesTex · 2026-11-12
- Scale AI paper: 36% of 374 attack runs succeed — frontier models can covertly sabotage their successors — Miles_Brundage · 2026-11-12
- Stanford ICML study featured on CNBC: AI agent teams suffer from groupthink — StanfordAILab · 2026-11-12
- Aaronson-Ambainis Conjecture proven; stronger variant also proved in appendix — TuhinChakr · 2026-11-12