StableReinforce Sparks a Score-Centering Wave in RL

Episode 1 · Tinker's StableReinforce Reverse-Engineered as REINFORCE with Score Centering (2026-10-25, 2 posts)

A community member reverse-engineered Tinker's new StableReinforce objective, revealing it is essentially REINFORCE with score centering, a finding that brought the score centering technique into the spotlight.

• Tinker's StableReinforce reverse-engineered: it's REINFORCE plus score centering — andersonbcdefg · 2026-10-25
• Reverse-engineering Tinker's StableReinforce: it's Reinforce + Score Centering — RichmanRonald · 2026-10-26

Episode 2 · TRL Makes Score Centering + Masked Importance Sampling Default for Async GRPO (2026-10-27, 5 posts)

Hugging Face's TRL team (@lewtun, @lvwerra) announced that score centering and masked importance sampling have been merged into TRL (PR #7760) and are now the default for the AsyncGRPO Trainer. This brings the method from Thinking Machines' paper "Score Centering Stabilizes Off-policy Reinforcement Learning" into a mainstream open-source RL training stack.

Confirmed

• TRL's AsyncGRPO Trainer enables score centering + masked importance sampling by default (PR #7760)
• @lvwerra confirmed the feature is stable; the underlying research shows LLM RL training becomes highly unstable when training and sampling policies are numerically inconsistent, and conventional fixes (numerics matching, importance sampling) only work around the problem
• @lewtun reported validation across various staleness and sampler quantization settings, including FP8 samplers no longer collapsing to 0.10
• The open-source RL framework slime reshared and recommended the update, suggesting async RL training recipes are converging

Why it matters

• Async/off-policy RL is a dominant direction for LLM training, and policy numeric mismatch is a core pain point; score centering targets the root cause rather than sidestepping it
• Shipping as a TRL default gives ordinary users stable training out of the box, and adoption across frameworks positions score centering to become a standard component of async RL

• Hugging Face TRL makes score centering + masked importance sampling default in async GRPO — _lewtun · 2026-10-27
• TRL adds Score Centering, a simple fix for RLHF training instability — lvwerra · 2026-10-27
• Score centering lands in TRL: default-on for async GRPO, stabilizes RL under mismatch — _lewtun · 2026-10-27
• TRL makes score centering the default for async GRPO: FP8 samplers now train like bf16 (0.72) — _lewtun · 2026-10-27
• Async RL recipes converge: score centering and masked importance sampling now default in TRL — _lewtun · 2026-10-27

Episode 3 · Off-policy drift emerges as the central problem in post-GRPO LLM RL research (2026-10-31, 3 posts)

Since GRPO's rise, the mismatch between training and inference policies has become the central issue in LLM RL research: the Score Centering paper proposes an additive correction to stabilize off-policy training, while a 48-page long-read surveys 14 correction methods.

• natolambert: importance-sampling fixes for train-inference mismatch dominate RL research — natolambert · 2026-10-31
• Score Centering paper: additive correction stabilizes off-policy RL under train-inference mismatch — natolambert · 2026-10-31
• 48-page deep dive: why modern LLM RL is secretly off-policy and 14 ways to fix it — cwolferesearch · 2026-10-31