New RL paper flags two common mistakes in KL-gradient estimation for LLM training
kastnerkyle · x · 2026-07-26
The paper points out several implementation pitfalls in estimating gradients for KL divergence during RL training for LLMs.
It highlights two main issues:
- Some common implementations differentiate through the KL estimate as if it were the loss to minimize, but that does not produce the intended KL gradient.
- Other implementations fail to account for the sequential nature of the estimation problem, which can lead to only a partial gradient.
The authors show the impact of these mistakes with tabular examples and LLM experiments, and they explain the correct way to implement the KL gradient. The post suggests this could affect practical open-source RLHF stacks such as TRL and Open Instruct, as well as pseudocode used in seminal work like GRPO.
Related event: New Research Reveals Flaws in KL Estimators for LLM RL Training(4 posts)→
More from Research
- A question probes how multi-agent branching scales against compute budget and model size — iskander · 2026-07-27
- NUS builds a soft force sensor that drives actuators without electronics or power — CurieuxExplorer · 2026-07-27
- Chelsea Finn says robot RL is bottlenecked by physical rollout cost, not algorithms — ycombinator · 2026-07-27
- ICML 2026 oral paper replication scores stay middling after a stricter re-scoring — profjamesevans · 2026-07-27
- Long-running agents will need immutable event logs, this thread argues — sebpaquet · 2026-07-27
- Seed IQ navigates Doom II, prompting questions about benchmarks beyond ARC-AGI — Fit_Transition8824 · 2026-07-27