Paper finds common KL estimators in LLM RL can produce the wrong gradients
burny_tech · x · 2026-07-26
- A paper on KL regularization in RL training of LLMs argues that common sample-based KL estimators can produce the wrong gradients.
- The authors show that the k3 estimator used in DeepSeek-style setups is unbiased for the value of expected reverse KL, but its gradient is not the gradient of KL; instead, it behaves like an unbiased estimator for the gradient of expected forward KL.
- Their empirical study finds that in on-policy settings, biased-gradient estimator choices can destabilize training, while unbiased-gradient configurations improve both in-domain and out-of-domain performance.
- They also note that KL regularization can help stabilize asynchronous training and provide practical takeaways for RL post-training of LLMs.
Related event: New Research Reveals Flaws in KL Estimators for LLM RL Training(4 posts)→
More from Research
- Qwen3.8-Flash-Next's open 180B weights hailed as a model organism for backdoor research — jd_pressman · 2026-10-27
- Latent-Foresight: world models that learn to predict their own future — abursuc · 2026-10-27
- UK AISI papers: no good safeguards yet against misuse of biological AI tools — davidmanheim · 2026-10-27
- CMU's XYBench: LLMs answer queries literally and reinforce user misconceptions — valentina__py · 2026-10-27
- arXiv advice from T. Dietterich: wrong primary category will almost surely get your paper on hold — tdietterich · 2026-10-27
- arXiv has 4,000+ papers on hold amid submission surge; moderator shares how to avoid it — tdietterich · 2026-10-27