Together AI: score centering stabilizes off-policy RL under train-inference mismatch

PandaAshwinee · x · 2026-09-18

A Together AI paper by Martin Marek & Max Ryabinin tackles train-inference mismatch (TIM) in LLM RL.

Problem: tiny differences between training and inference engines destabilize RL; fully eliminating TIM would cripple rollout efficiency
Diagnosis: instability is primarily caused by drift — a persistent bias between engines that accumulates every training step
Method: an additive "score centering" correction term that cancels drift
Results: on models from 0.6B to 30B parameters, score centering alone matches or outperforms importance sampling (TIS) under quantization, with the gap growing as mismatch worsens
Composition: since it's additive, it stacks with importance sampling, beating pure IS baselines in staleness experiments
Experiments train Qwen3-30B-A3B-Base on INTELLECT-2 math data under progressively heavier sampler quantization

More from Research