Score centering: the simple fix for LLM RL instability under training-inference mismatch
cwolferesearch · x · 2026-11-10
Cameron Wolfe published a deep dive on score centering, a technique (allegedly) popular in frontier labs for stabilizing LLM RL.
Core problem: training-inference mismatch (TIM)
- Modern RL infra splits rollout generation (sampler) and policy updates (trainer) into two engines that should represent the same policy, but numerical precision, quantization, and different GPU kernels cause their token probabilities to diverge.
- This mismatch makes RL increasingly unstable as models scale to large MoEs, disaggregated/async architectures, and agentic tasks.
The fix
- Score centering cleanly explains why TIM breaks the log-derivative trick and causes gradient drift.
- It applies a simple additive correction to the policy gradient: center the score by subtracting its expected value under the sampler—a correction that depends only on the next token and is identical for every token, canceling the sampler-trainer ratio bias.
The post includes efficient implementations, combinations with importance sampling, and Qwen3 training curves showing markedly more stable accuracy under weight noise and stale rollouts.
More from Infra
- Data-center water use debate: a post misunderstanding basic water science gets 22K likes — ATTlKA · 2026-11-10
- Qdrant presents vector index-native relevance feedback in production at CIKM Rome — qdrant_engine · 2026-11-10
- int8 quantization plus fp16 rescoring: 1.4M embeddings retrieved in under 2ms — tomaarsen · 2026-11-10
- int8 embeddings plus Usearch index: 1.4M vectors searched in under 2ms on CPU — tomaarsen · 2026-11-10
- Google rolls out broader AI chip lineup to challenge Nvidia in a market worth hundreds of billions — dinabass · 2026-11-10
- Thinking Machines engineer returns to Modal, reflecting on a year of frontier ML infrastructure work — josh_wills · 2026-11-10