RLVR preprint finds only one reverse-KL estimator stays unbiased in LLM fine-tuning
heghbalz · x · 2026-07-26
A new preprint evaluates which estimator to use for the reverse-KL term in RLVR fine-tuning of LLMs. The study compares common choices, including K1 in the reward and K3 in the loss, and finds that every configuration except K1 in the reward produces biased gradients.
The authors also report that two biased setups lead to training instability or collapse during RL fine-tuning. Their table summarizes the estimator formulas, whether the gradient estimate is unbiased, and the observed training behavior, making the paper a practical guide for RLVR implementations.
Related event: New Research Reveals Flaws in KL Estimators for LLM RL Training(4 posts)→
More from Research
- Kimi K3 may be strong on cyber, but token efficiency keeps it off UK AISIS — teortaxesTex · 2026-07-27
- ARC AGI 3 should have stayed private, with no examples or public dataset — flowersslop · 2026-07-27
- ExploitGym may have only 60–70% solvable tasks, fueling the OpenAI cheating debate — max_paperclips · 2026-07-27
- RTX 5090 local tests show Qwen Q6 can drop to 15 tok/s at 80k context — LFAdvice7984 · 2026-07-27
- Noahpinion quotes Chollet: intelligence may hit a hard ceiling — binarybits · 2026-07-27
- Paper argues graph topology can become the core operating system for AI agents — theomitsa · 2026-07-27