Paper finds common KL estimators in LLM RL can produce the wrong gradients

burny_tech · x · 2026-07-26

A paper on KL regularization in RL training of LLMs argues that common sample-based KL estimators can produce the wrong gradients.
The authors show that the k3 estimator used in DeepSeek-style setups is unbiased for the value of expected reverse KL, but its gradient is not the gradient of KL; instead, it behaves like an unbiased estimator for the gradient of expected forward KL.
Their empirical study finds that in on-policy settings, biased-gradient estimator choices can destabilize training, while unbiased-gradient configurations improve both in-domain and out-of-domain performance.
They also note that KL regularization can help stabilize asynchronous training and provide practical takeaways for RL post-training of LLMs.

More from Research