Zeroing train-infer mismatch lifts open-source RL solve rate from 63.9% to 77.4%
nrehiew_ · x · 2026-08-19
A technical blog post (with open-source code) tackles train-infer mismatch, endemic in open-source RL stacks: the inference engine (SGLang, vLLM, TensorRT-LLM) samples rollouts while the training engine (Megatron-LM, TorchTitan, XoRL) recomputes logprobs, and the two can disagree. Mismatch arises from floating-point reduction order plus architecture-specific issues in GDN and sparse attention.
Key result: training Qwen3.6-35B-A3B to play Wordle, their XoRL engine aligned with a forked SGLang to produce bitwise-identical logprobs (0 mismatch), lifting solve rate from 63.9% to 77.4%. Closed-source managed RL services River and Tinker, both using vanilla importance sampling, show similar measured mismatch (K3) yet clearly different solve rates; Tinker does much better trained with CISPO.
The article covers the costs and benefits of 0-mismatch training, alternative mitigations, and ablations. All code is open-source, applicable to open-weight MoEs like DeepSeek, GLM and Qwen.
Related event: Fixing Train-Infer Mismatch Boosts Open-Source RL Performance(2 posts)→
More from Infra
- Cerebras announces new AI accelerator CS-4 — scaling01 · 2026-08-19
- 16-bit Model Requires 60GB VRAM; 4-bit Quantized Fits on 24GB — LeviTurk · 2026-08-19
- Why Removing the Vision Encoder Can Be Better — From an Infra Perspective — liuziwei7 · 2026-08-19
- Matmul Optimization Bottleneck: Data Movement, Not Multiplications — yaroslavvb · 2026-08-19
- China opens world's largest AI data center targeting 1 million GPUs — teortaxesTex · 2026-08-19
- Morgan Stanley: US data centers need 68GW power by 2028 — tctjr · 2026-08-19