RL-Kernel achieves bitwise-identical train/rollout logprobs over 200 GRPO steps
青稞AI · wechat · 2026-10-18
The hidden numerical trap in RL post-training
In large-model RL training, rollout and training typically run on two different engines: training is optimized for backprop, sequence packing, and multi-GPU parallelism, while inference is optimized for prefill/decode, dynamic batching, and paged KV cache. Even with identical weights and tokens, differences in floating-point reduction order, parallel sharding, and precision-conversion boundaries can cause the LogP values to diverge (e.g., 0.1+0.2+0.3 vs 0.3+0.1+0.2 may differ under finite precision).
In PPO/GRPO, the importance ratio is computed from rollout-side and training-side logprobs; any nonzero bias acts as an extra numerical policy drift, propagating through KL diagnostics and clipping, often showing up as growing instability over training.
The RL-Kernel + vLLM approach
- vime aligns the timeline: which token batch belongs to which step and weight version, and when new weights sync to vLLM.
- RL-Kernel aligns the numerical execution: reduction-domain partitioning, partial-result merge order, intermediate precision, and rounding points — making both engines follow the same numerical execution contract. Engineering work includes per-op alignment of attention, FFN, logprob, and communication paths, plus new deterministic MFMA GEMM kernels, vocabulary reduction, and HIP IPC on ROCm.
Results
On AMD Instinct MI300X with 8 GPUs running Qwen3-8B GRPO end-to-end for 200 consecutive steps: mismatchcount=0 and maxabsdiff=0 — bitwise-consistent logprobs between training and rollout throughout.
Qingke community will host RL-Kernel core member and XJTU master's student Li Jiajie on Oct 20, covering how to localize train/inference divergence, numerical contract alignment, fixed-replay and layer-by-layer comparison validation, and consistency boundaries under PD disaggregation and async RL.
More from Infra
- SGLang Summit sponsors grow from 11 to 36 in one week; Lip-Bu Tan, Lilian Weng to speak — BanghuaZ · 2026-10-18
- Yacine jokes his datacenter KPI is gigawatt-hours: get paged if not burning enough power — yacineMTB · 2026-10-18
- OpenAI reportedly handled ~15T tokens daily in April, likely higher now — JoshPurtell · 2026-10-18
- 1.68x inference speedup drops, directly accelerating GRPO post-training — yacineMTB · 2026-10-18
- OpenRouter is only a fraction of the total addressable market, poster argues — sloppenheimer · 2026-10-18
- Developer builds change alerts for LLM prices and retirements across Bedrock, Azure and Vertex — jsandhu0780 · 2026-10-18