vLLM open-sources agent skill for bitwise-identical train-inference logprobs in RL
vllm_project · x · 2026-10-15
The vLLM team distilled their debugging workflow into an open-source agent skill (PR #61551), aiming for bitwise identical logprobs between RL training and vLLM rollout — true on-policy RL — while keeping rollout throughput close to the high-performance baseline.
The core is a C0–C10 gated workflow with explicit pass criteria per stage: from module inventory and a proxy model, through inference batch / prefill-decode invariance, training forward & backward alignment, two-step RL acceptance, to delivery. Guides cover mismatch localization and bisecting, training forward/backward validation, and which batch-invariant kernel each operator should use. The loop: find the first diverging op → minimal repro → fix → re-verify → profile and optimize.
Results on Nemotron-3.5-Lightning-30B-A3B (Mamba-2 hybrid with NVFP4): on a filtered SWE-bench Pro training subset, train–inference KL stays exactly 0; training reward rises 0.56 → 0.59 while the unpatched baseline falls 0.56 → 0.22; rollout throughput is 87.5% of baseline over the first 30 steps. DeepSeek-V4.1 has also reached 0 diff. Caveats: each baseline used one R3 config without TIS/MIS and was not thoroughly tuned; end-to-end training wall-clock speedup remains to be measured.
More from coding & agent
- Marc Lou praises PostShiba, a cute indie alternative to AWS SES for email infra — marclou · 2026-10-16
- RExBench: best coding agent completes only 33% of autonomous research extensions — najoungkim · 2026-10-16
- Wiring Pstack skills from a Grok bot into Hermes agent loops — alexcovo_eth · 2026-10-16
- past.dev launches long-term memory API for AI agents, tops BEAM public benchmark — thisiskp_ · 2026-10-16
- Google's CLASHarness benchmarks multi-agent conflicts in computer use, finds frontier VLMs struggle — google · 2026-10-16
- mitsuhiko's AI agent solved multiplayer with PeerJS before he could explain it — mitsuhiko · 2026-10-16