RLVR preprint finds only one reverse-KL estimator stays unbiased in LLM fine-tuning

heghbalz · x · 2026-07-26

A new preprint evaluates which estimator to use for the reverse-KL term in RLVR fine-tuning of LLMs. The study compares common choices, including K1 in the reward and K3 in the loss, and finds that every configuration except K1 in the reward produces biased gradients.

The authors also report that two biased setups lead to training instability or collapse during RL fine-tuning. Their table summarizes the estimator formulas, whether the gradient estimate is unbiased, and the observed training behavior, making the paper a practical guide for RLVR implementations.

Original post →

RLVR preprint finds only one reverse-KL estimator stays unbiased in LLM fine-tuning

More from Research