Paper finds common KL estimators in LLM RL can produce the wrong gradients
burny_tech · x · 2026-07-26
- A paper on KL regularization in RL training of LLMs argues that common sample-based KL estimators can produce the wrong gradients.
- The authors show that the k3 estimator used in DeepSeek-style setups is unbiased for the value of expected reverse KL, but its gradient is not the gradient of KL; instead, it behaves like an unbiased estimator for the gradient of expected forward KL.
- Their empirical study finds that in on-policy settings, biased-gradient estimator choices can destabilize training, while unbiased-gradient configurations improve both in-domain and out-of-domain performance.
- They also note that KL regularization can help stabilize asynchronous training and provide practical takeaways for RL post-training of LLMs.
Related event: New Research Reveals Flaws in KL Estimators for LLM RL Training(4 posts)→
More from Research
- A question probes how multi-agent branching scales against compute budget and model size — iskander · 2026-07-27
- NUS builds a soft force sensor that drives actuators without electronics or power — CurieuxExplorer · 2026-07-27
- Chelsea Finn says robot RL is bottlenecked by physical rollout cost, not algorithms — ycombinator · 2026-07-27
- ICML 2026 oral paper replication scores stay middling after a stricter re-scoring — profjamesevans · 2026-07-27
- Long-running agents will need immutable event logs, this thread argues — sebpaquet · 2026-07-27
- Seed IQ navigates Doom II, prompting questions about benchmarks beyond ARC-AGI — Fit_Transition8824 · 2026-07-27