New RL paper flags two common mistakes in KL-gradient estimation for LLM training
kastnerkyle · x · 2026-07-26
The paper points out several implementation pitfalls in estimating gradients for KL divergence during RL training for LLMs.
It highlights two main issues:
- Some common implementations differentiate through the KL estimate as if it were the loss to minimize, but that does not produce the intended KL gradient.
- Other implementations fail to account for the sequential nature of the estimation problem, which can lead to only a partial gradient.
The authors show the impact of these mistakes with tabular examples and LLM experiments, and they explain the correct way to implement the KL gradient. The post suggests this could affect practical open-source RLHF stacks such as TRL and Open Instruct, as well as pseudocode used in seminal work like GRPO.
Related event: New Research Reveals Flaws in KL Estimators for LLM RL Training(4 posts)→
More from Research
- EdgeRock-4B holds KV cache at 72 MiB across 256K context, 99.1% smaller than Qwen3.5-4B — alifcoder · 2026-10-27
- TRL adds Score Centering, a simple fix for RLHF training instability — lvwerra · 2026-10-27
- Latent-Foresight: world models that learn to predict their own future — abursuc · 2026-10-27
- UK AISI papers: no good safeguards yet against misuse of biological AI tools — davidmanheim · 2026-10-27
- CMU's XYBench: LLMs answer queries literally and reinforce user misconceptions — valentina__py · 2026-10-27
- arXiv advice from T. Dietterich: wrong primary category will almost surely get your paper on hold — tdietterich · 2026-10-27