Score centering and importance sampling are complementary fixes for training-inference mismatch in LLM RL
cwolferesearch · x · 2026-11-14
Cameron Wolfe breaks down why score centering and importance sampling address different halves of training-inference mismatch (TIM) in LLM RL:
- The problem: tokens are sampled by one policy while gradients use another, creating a spurious 'gradient drift' in the policy gradient.
- Score centering: subtracts the expected score under the sampler, additively removing drift while preserving reward-dependent covariance — but learning signal is still measured under the sampler distribution.
- Importance sampling: reweights contributions toward the trainer distribution, but ratios can explode; TIS/MIS clipping reintroduces drift.
- Combined: importance-weight the scores, then center them — solving both drift and mismatch without unbounded ratios.
- Empirics: score centering alone suffices for mild mismatch (e.g., quantization); under severe mismatch (stale rollouts, INT8 sampler), SC + TIS/MIS clearly wins.
Takeaway: the combination is especially valuable for asynchronous RL where the sampler lags the trainer.
Related event: Researchers explain complementary fixes for LLM RL train-inference mismatch(2 posts)→
More from Research
- François Fleuret asks: is the gradient Japanese? It never refuses to give you a direction — francoisfleuret · 2026-11-22
- Hat Matrix Explained: How Much Leverage Does Each Data Point Have in Least Squares? — RobertTLange · 2026-11-22
- Energy-Based Reasoners: gradient descent on latent states for adaptive reasoning — kalyan_kpl · 2026-11-22
- Lean community certifies Riemann proof; honeypot fake keys expose breaches at three AI providers — LeviTurk · 2026-11-22
- Riemann hypothesis 'proven by Turkish mathematician using AI' claim mocked as the 'Organic Turk' — CatAstro_Piyush · 2026-11-22
- Andriy Burkov releases The Hundred-Page Language Models Book, hands-on with PyTorch — burkov · 2026-11-22