RL KL estimators may optimize the wrong direction, and k1 can collapse to zero gradient
auto_grad_ · x · 2026-07-25
- The post argues that common KL estimators used in RL losses can have incorrect gradients for the objective they are supposed to optimize.
- In particular, it claims k3 (used in DeepSeek series) is an unbiased estimator of the value of reverse KL, but its gradient is actually an unbiased estimator of the gradient of forward KL instead.
- It also points out that the original k1 estimator, the raw log-ratio, has an expectation whose gradient is zero — meaning it is effectively optimizing nothing.
- The screenshots walk through the derivations and show why the gradients line up with the opposite KL direction or collapse to zero.
Related event: New Research Reveals Flaws in KL Estimators for LLM RL Training(4 posts)→
More from Research
- What makes research get adopted: important problems, proper evals, and usability — srchvrs · 2026-10-27
- Bitter lesson part 2: few academic inventions matter, and conference publications aren't the key — srchvrs · 2026-10-27
- The four criteria for research impact: important problems, strong baselines, scalability, real usability — srchvrs · 2026-10-27
- EMNLP keynote: industry does read academic NLP papers, it just rarely credits them — srchvrs · 2026-10-27
- Researcher says GitHub used his team's bit-packing work but credited only 1 of 3 authors — srchvrs · 2026-10-27
- First-week PhD student lands ARR Area Chair role, exposing peer-review quirks — anmarasovic · 2026-10-27