REINFORCE-Zero: kernel alignment + unclipped IS enables 20K-step stable async RL
青稞AI · wechat · 2026-11-06
The REINFORCE-Zero project reports a first for the open-source community: by aligning trainer and inference numerical execution (Trainer–Inference Invariance, TII), plain unclipped token-level importance sampling suffices for stable asynchronous LLM RL for 20,000 policy updates — no ratio clipping, no rejection gates.
Core insight
- Asynchronous bias splits into weight lag (correctable by ordinary IS) and kernel mismatch from batch composition, prefill/decode paths, and float ordering (which IS cannot handle)
- With standard kernels, IS diagnostic peak/median ratio hit 442.54; after alignment it dropped to 1.50, and lag-2/lag-8 runs trained to 20K steps reaching 51.10%/50.93% average pass@1 on five math benchmarks
Results
- High-latency tool calling (Qwen3-4B, lag 64): reward 0.287 → 0.782
- 30B MoE (Qwen3-30B-A3B): average pass@1 31.29% → 71.55%
- OSWorld visual computer use: 50.97% → 56.16%
- Same 16 GPUs: async training step time cut from 164.77h to 68.55h, a 2.40x throughput gain
The kernel-alignment workflow is packaged as a reusable agent skill (11 checks, C0-C10) used with Claude Code to wire vLLM's batch-invariant kernels into AutoModel. Paper, code, and skill are all open-sourced.
Related event: REINFORCE Zero: Open-Source Asynchronous LLM RL Runs Stably for 20K Steps(3 posts)→
More from coding & agent
- Multi-model agent setup: letting Opus delegate to Astra works better than expected — gandamu_ml · 2026-11-06
- Supply-chain vector in almost every codebase: unpinned GitHub Actions can change under you — andersonbcdefg · 2026-11-06
- Chalk talk: Jediah Katz gives an inside look at their agent harness design — jxnlco · 2026-11-06
- Tinker team benchmarks model finetunability: Inkling 2 Small learns faster for less — ziqiao_ma · 2026-11-06
- Qwen 27B + Python REPL harness hits 78.9% on Harvey benchmark, beating larger flagships — bansalg_ · 2026-11-06
- One Slack ask, 34 merged PRs in under 6 hours: Sentry CEO on running coding agents — zeeg · 2026-11-06