New RL paper flags two common mistakes in KL-gradient estimation for LLM training
kastnerkyle · x · 2026-07-26
The paper points out several implementation pitfalls in estimating gradients for KL divergence during RL training for LLMs.
It highlights two main issues:
- Some common implementations differentiate through the KL estimate as if it were the loss to minimize, but that does not produce the intended KL gradient.
- Other implementations fail to account for the sequential nature of the estimation problem, which can lead to only a partial gradient.
The authors show the impact of these mistakes with tabular examples and LLM experiments, and they explain the correct way to implement the KL gradient. The post suggests this could affect practical open-source RLHF stacks such as TRL and Open Instruct, as well as pseudocode used in seminal work like GRPO.
Related event: New Research Reveals Flaws in KL Estimators for LLM RL Training(4 posts)→
More from Research
- 3D ResNet Paper Crosses 3,000 Citations Eight Years After CVPR 2018 — HirokatuKataoka · 2026-09-11
- Sample selection and ordering matter a lot in LLM training: DataFlex makes data scheduling dynamic — Puzzleheaded_Box2842 · 2026-09-11
- Jeff Heaton's Intro to the Math of Neural Networks eBook Is Free to Download — blaizedsouza · 2026-09-11
- Mathematician Daniel Litt Launches Problem Repo to Track Human vs AI Progress: 15 Problems, 1 Solved — littmath · 2026-09-11
- Open ECDSA.fail challenge uses AI agents to shrink Shor's-algorithm quantum circuits for Bitcoin keys — StefanoGogioso · 2026-09-11
- Alex Townsend posts 200 open problems in numerical linear algebra for humans and AI agents — IgorCarron · 2026-09-11