Together AI's score centering fixes RL training collapse under quantized inference
nrehiew_ · x · 2026-10-16
Together AI researchers Martin Marek and Max Ryabinin released a paper showing that instability in LLM RL under training-inference mismatch (TIM) is caused by drift — a persistent bias between training and inference engines. They derive an additive "score centering" correction term that cancels this drift.
Key findings:
- For models from 0.6B to 30B, score centering alone matches or outperforms importance sampling under aggressive quantization (FP8/INT4 weights and KV cache)
- The heavier the inference quantization, the wider the gap in favor of score centering
- It composes with importance sampling, outperforming pure IS
On Qwen3-30B-A3B trained against INTELLECT-2 math data, score centering stayed stable over 800 steps while naive PG collapsed. Author nrehiew thinks it will become part of the standard RL recipe by default.
More from Infra
- TokenSpeed MLA 2 hits Vera Rubin: up to 3.44× faster MLA decode for Kimi K3 — zhyncs42 · 2026-10-16
- US vs Chinese hyperscalers: Americans fund and schmooze, Chinese demand your BoM and cap you at 7% margin — jwt0625 · 2026-10-16
- How LiteLLM routes one OpenAI-shaped API across every model provider — Mahmoud_Zalt · 2026-10-16
- OpenAI-Broadcom custom AI chip Jalapeño named a TIME Best Invention of 2026 — itsclivetime · 2026-10-16
- Developer live-runs continual pretraining of a Luganda LLM on Colab, 41% done with loss dropping fast — weskambale · 2026-10-16
- Jensen Huang: Nvidia charges ~$thousands per GPU yearly for software, a major revenue line — rohanpaul_ai · 2026-10-16