Score centering lands in TRL: default-on for async GRPO, stabilizes RL under mismatch
_lewtun · x · 2026-10-27
The Thinking Machines paper "Score Centering Stabilizes Off-policy Reinforcement Learning" is now integrated in Hugging Face TRL and enabled by default in the Async GRPO Trainer (alongside masked importance sampling).
Key points from the paper:
- RL for LLMs is notoriously sensitive to training-inference mismatch (TIM); eliminating TIM entirely comes at a major cost to rollout efficiency
- The authors show instability under TIM is primarily caused by drift: a persistent bias between training and inference engines that accumulates every step
- They derive an additive "score centering" correction term that cancels drift and stabilizes training
- On 0.6B–30B models, score centering alone matches or beats importance-sampling methods under quantization, with the gap growing as mismatch worsens
- Being additive, it composes with importance sampling; the combination outperforms pure IS baselines in staleness experiments
QGallouedec reports the stability across staleness and sampler-quantization settings is "pretty wild".
Related event: TRL Async GRPO Trainer Enables Score Centering by Default(3 posts)→
More from coding & agent
- Mitsuhiko Cleans Up the AI-Made Game He Built With His Kids — mitsuhiko · 2026-10-27
- 'Taste skills' are AI's biggest lie — code is commoditized, your eye is the moat — aryanXmahajan · 2026-10-27
- Agent Buddy: A Desk Gadget That Shows What Your Background AI Coding Agents Are Doing — DanWahlin · 2026-10-27
- kimo: open-source GUI-less video editor lets AI agents cut video via typed MCP tools and render frames to self-verify — LieTop7965 · 2026-10-27
- Weekend RAG turns into a full-time job; Stack Overflow's Stack Internal absorbs the cost — pchandrasekar · 2026-10-27
- Should Terra replace Sol as the default Codex model? — petergyang · 2026-10-27