Score Centering paper: additive correction stabilizes off-policy RL under train-inference mismatch
natolambert · x · 2026-10-31
The arXiv paper "Score Centering Stabilizes Off-policy Reinforcement Learning" (Martin Marek, Max Ryabinin):
- Problem: LLM RL is notoriously sensitive to train-inference mismatch (TIM), but fully eliminating TIM would heavily cost rollout efficiency
- Finding: instability under TIM is mainly caused by drift — a persistent bias between training and inference engines accumulating every step
- Method: an additive "score centering" correction term cancels drift and stabilizes RL
- Results: training 0.6B–30B models, score centering alone matches or beats importance-sampling methods under quantization, with the gap growing as mismatch worsens; being additive, it composes with IS and outperforms pure IS baselines in staleness experiments
Nathan Lambert highlighted the paper's appendix table comparing IS-based methods (TIS/CISPO, MIS/IcePop, PPO, DAPO, GSPO, TOPR, DPPO) — the biggest focus of RL algorithm research since GRPO's takeoff.
More from Research
- AA-Omniscience v1.1: public set matches private benchmark at 0.99 correlation — ArtificialAnlys · 2026-10-31
- AA-Omniscience v1.1 launches: Gemini 4 Carbon tops knowledge benchmark as some models drop 20+ points — ArtificialAnlys · 2026-10-31
- Santa Fe Institute publishes interactive history of agent-based modeling — round · 2026-10-31
- Peking University's DragMesh turns static 3D assets into articulated objects with one drag, accepted to IJCV — jiqizhixin · 2026-10-31
- Higher reasoning effort doesn't help AI agents catch silent failures, τ-bench test finds — TruthDeep8659 · 2026-10-31
- 'The Proof in the Code' named among 2026's best nonfiction: the story of Lean and human-AI collaboration — rbhar90 · 2026-10-31