RLVR preprint finds only one reverse-KL estimator stays unbiased in LLM fine-tuning
heghbalz · x · 2026-07-26
A new preprint evaluates which estimator to use for the reverse-KL term in RLVR fine-tuning of LLMs. The study compares common choices, including K1 in the reward and K3 in the loss, and finds that every configuration except K1 in the reward produces biased gradients.
The authors also report that two biased setups lead to training instability or collapse during RL fine-tuning. Their table summarizes the estimator formulas, whether the gradient estimate is unbiased, and the observed training behavior, making the paper a practical guide for RLVR implementations.
Related event: New Research Reveals Flaws in KL Estimators for LLM RL Training(4 posts)→
More from Research
- EdgeRock open-sourced: 72 MiB KV cache at 256K context, 99.1% smaller than Qwen3.5-4B — alifcoder · 2026-10-27
- EdgeRock-4B holds KV cache at 72 MiB across 256K context, 99.1% smaller than Qwen3.5-4B — alifcoder · 2026-10-27
- TRL adds Score Centering, a simple fix for RLHF training instability — lvwerra · 2026-10-27
- Latent-Foresight: world models that learn to predict their own future — abursuc · 2026-10-27
- UK AISI papers: no good safeguards yet against misuse of biological AI tools — davidmanheim · 2026-10-27
- CMU's XYBench: LLMs answer queries literally and reinforce user misconceptions — valentina__py · 2026-10-27