Paper: additive "score centering" term stabilizes LLM RL under training-inference mismatch
finbarrtimbers · x · 2026-10-25
New arXiv paper Score Centering Stabilizes Off-policy Reinforcement Learning (Martin Marek, Max Ryabinin) tackles the training-inference mismatch (TIM) problem in RL for LLMs.
- Key finding: RL instability under TIM is mainly caused by drift—a persistent bias between training and inference engines that accumulates each step.
- Method: an additive "score centering" correction term that cancels drift.
- Results: on models from 0.6B to 30B, score centering alone matches or beats importance-sampling-based methods under quantization, with the gap widening as mismatch grows; being additive, it composes with IS and outperforms pure IS baselines in staleness experiments.
- Significance: fully eliminating TIM is impractical due to rollout efficiency costs; this correction offers a cheap way around the trade-off.
More from Research
- "Do LLMs Feel?" paper maps localized emotion circuits inside LLMs — mikeflache · 2026-10-25
- Researchers push back on hype that AI papers done in 2–3 days make peer review obsolete — ChongZzZhang · 2026-10-25
- EC2-Bench: edge-cloud collaboration hits 73% of cloud-only agent performance at 15% of the cost — zibuyu9 · 2026-10-25
- Score centering cancels RL training-inference drift via a bias term — finbarrtimbers · 2026-10-25
- TensorFold's experimental Living Weights writes new facts into local model weights — GabGarrett · 2026-10-25
- Researcher argues AI tooling should filter junk papers so scientists focus on work worth their time — hugo_larochelle · 2026-10-25