Score centering and importance sampling are complementary fixes for training-inference mismatch in LLM RL

cwolferesearch · x · 2026-11-14

Colin Fraser explains training-inference mismatch (TIM) in LLM RL: tokens are sampled by one policy while gradients are computed under another, so the expected token-level score under the sampler is nonzero, introducing bias.

He argues the two approaches address TIM differently and are complementary rather than competing, with details in his blog post.

Related event: Researchers explain complementary fixes for LLM RL train-inference mismatch(2 posts)→

Original post →

More from Research

Research channel →