Together AI: score centering stabilizes off-policy RL under train-inference mismatch
PandaAshwinee · x · 2026-09-18
A Together AI paper by Martin Marek & Max Ryabinin tackles train-inference mismatch (TIM) in LLM RL.
- Problem: tiny differences between training and inference engines destabilize RL; fully eliminating TIM would cripple rollout efficiency
- Diagnosis: instability is primarily caused by drift — a persistent bias between engines that accumulates every training step
- Method: an additive "score centering" correction term that cancels drift
- Results: on models from 0.6B to 30B parameters, score centering alone matches or outperforms importance sampling (TIS) under quantization, with the gap growing as mismatch worsens
- Composition: since it's additive, it stacks with importance sampling, beating pure IS baselines in staleness experiments
- Experiments train Qwen3-30B-A3B-Base on INTELLECT-2 math data under progressively heavier sampler quantization
Related event: Together AI Proposes Score Centering to Fix RL Train-Inference Mismatch(2 posts)→
More from Research
- Researcher slams conference rule banning post-acceptance paper revisions at ICSE — tianyin_xu · 2026-09-18
- Google's learning interactives library hits 36 exercises; only 3.5% pass checks first try — xiaohu · 2026-09-18
- Visual General Intelligence white paper authors to present at Video Model Journal Club — HirokatuKataoka · 2026-09-18
- The subtle KV cache detail most people get wrong: tokens generate before entering the cache — techNmak · 2026-09-18
- You can turn jev back into an autoregressive LLM just by making it output ASCII — multiply_matrix · 2026-09-18
- Follow-up: December paper claims 8.1 Mbps peak throughput, but major physics hurdles remain — BlancheMinerva · 2026-09-18