Together AI's Score Centering Stabilizes Off-policy RL with One Correction Term
Together AI researchers Martin Marek and Max Ryabinin released the paper "Score Centering Stabilizes Off-policy Reinforcement Learning," proposing a principled correction called Score Centering for the instability caused by training-inference mismatch (TIM) in LLM reinforcement learning. As @AndrewZhao's thread highlights, the fix rests on nothing more than an undergraduate covariance identity, yet directly cancels the root cause of instability.
Confirmed
- The paper identifies the root cause of instability under TIM as drift: discrepancies between training and inference engines (numerical precision, different sampling frameworks) accumulate over training steps and eventually break stability.
- The correction uses the identity E[AB]=E[A]E[B]+Cov(A,B), adding a score-centering term to the loss that cancels the source of instability from first principles rather than working around it.
- @AndrewZhao notes that standard remedies such as numeric alignment and importance sampling only bypass the problem; the new method claims direct cancellation.
Why it matters
- Training-inference mismatch is a common pain point in LLM RL, usually mitigated with engineering-level numeric alignment. If effective, a correction term requiring only undergraduate probability could substantially loosen the strict requirement for engine consistency in RL training.
- The work demonstrates a path of analyzing and eliminating instability sources from first principles, offering a directly adoptable scheme for future RL training frameworks.
2026-09-18 ~ 2026-09-19 · 5 related posts
Primary sources
- Together AI: score centering stabilizes off-policy RL under train-inference mismatch — PandaAshwinee ·
- Together AI paper: score centering cancels drift in off-policy RL under train-inference mismatch — PandaAshwinee ·
- Score Centering tackles the root cause of RL instability when train and sample policies differ — _AndrewZhao ·
- [source] Together AI paper: score centering cancels drift in off-policy RL under train-inference mismatch — PandaAshwinee · 2026-09-18
- [source] Score Centering tackles the root cause of RL instability when train and sample policies differ — _AndrewZhao · 2026-09-19
- Score Centering fixes unstable RL for LLMs using an undergrad covariance identity — burny_tech · 2026-09-19
- Stabilizing RL with a simple second-year probability trick: the covariance identity — PMinervini · 2026-09-19
1 near-duplicate retellings: PandaAshwinee