vLLM Achieves Bitwise-Consistent RL Training, Fixing KL Blowups and Score Collapse

vllm_project · x · 2026-10-15

vLLM published a detailed blog on numerical consistency in RL training.

Problem: inference-side optimizations in high-performance RL (low-precision compute, FP8 KV cache, kernel fusion) change numerics, so training optimizes a policy other than the one that sampled. On DeepSeek-V4-Flash, the baseline's train-inference KL rose from 2e-3 to 29.9 while reward fell.
Approach: the training forward pass calls vLLM's inference kernels directly, modified so rollout and training produce bitwise-identical logprobs (diff = 0) for every sampled token, while retaining high-performance paths (FP8 GEMM, Top-K, FP8 KV cache, MoE communication, CUDA Graph).
Results: over 100 DeepSeek-V4-Flash steps, diff = 0 at every step at 91.3% of baseline rollout throughput. On Nemotron-3.5 (Mamba hybrid + NVFP4), the consistent run holds 0.6 on SWE-bench Pro while the baseline drops from 0.56 to 0.22.
Open source: packaged as an agent skill with recipes and code; DeepSeek-V4.1 has reached 0 diff, with performance work to follow in a separate post.

More from Infra