Score centering and importance sampling are complementary fixes for training-inference mismatch in LLM RL
cwolferesearch · x · 2026-11-14
Colin Fraser explains training-inference mismatch (TIM) in LLM RL: tokens are sampled by one policy while gradients are computed under another, so the expected token-level score under the sampler is nonzero, introducing bias.
- Score centering removes the systematic offset by centering the score
- Importance sampling corrects the distribution gap between the two policies
He argues the two approaches address TIM differently and are complementary rather than competing, with details in his blog post.
Related event: Researchers explain complementary fixes for LLM RL train-inference mismatch(2 posts)→
More from Research
- François Fleuret asks: is the gradient Japanese? It never refuses to give you a direction — francoisfleuret · 2026-11-22
- Hat Matrix Explained: How Much Leverage Does Each Data Point Have in Least Squares? — RobertTLange · 2026-11-22
- Energy-Based Reasoners: gradient descent on latent states for adaptive reasoning — kalyan_kpl · 2026-11-22
- Lean community certifies Riemann proof; honeypot fake keys expose breaches at three AI providers — LeviTurk · 2026-11-22
- Riemann hypothesis 'proven by Turkish mathematician using AI' claim mocked as the 'Organic Turk' — CatAstro_Piyush · 2026-11-22
- Andriy Burkov releases The Hundred-Page Language Models Book, hands-on with PyTorch — burkov · 2026-11-22