Paper: RL Causes "Coverage Collapse"—LLMs Forget Their Best Answers as Reward Rises
burny_tech · x · 2026-11-14
An alphaXiv thread covers "A Theory of Policy Optimization in Post-Training: Coverage Collapse and Reference Resets." Counterintuitive finding: RL can make an LLM forget how to generate its best answers even while average reward keeps improving.
The paper names this coverage collapse: policy optimization slowly reduces the probability of generating optimal responses, making them increasingly hard to rediscover during training.
The fix proposed is APMD (Approximate Policy Mirror Descent), which repeatedly resets the KL reference to the latest model instead of always anchoring to the original base model—enabling controlled improvements while continuing toward better solutions. Reference resets also give stronger convergence guarantees for simplified PPO, GRPO, and Kimi-style algorithms.
More from Research
- LLM-extracted chemistry rules find new crystals from just 22 examples — bravo_abad · 2026-11-14
- Terence Tao: math has moved from proof scarcity to proof abundance — haider1 · 2026-11-14
- Cmpnd releases dspy-wsp: $5-10 of GPT-6 edits model weights, lifts every decision model tested — dbreunig · 2026-11-14
- Mentored team wins 1st prize at Africa Big Data School with pulmonary fibrosis ML model — lamechthinkbig · 2026-11-14
- Li Bojie open-sources 'Understanding AI Infra' book, 7,500+ GitHub stars in weeks — udmrzn · 2026-11-14
- Russ Tedrake's CoRL talk: coupling deployment data with large-scale pretraining — RussTedrake · 2026-11-14