REINFORCE Zero: trainer-inference invariance keeps async LLM RL stable for 20,000 updates
teortaxesTex · x · 2026-11-05
- Researchers present REINFORCE Zero: with Trainer–Inference Invariance (TII) and a minimal importance-sampling ratio, plain REINFORCE suffices for stable LLM RL.
- The recipe uses unclipped token-level IS ratios, no rejection gates, and a minimal loss.
- Authors claim it's the first open-source 20K-update demo of this recipe: five-benchmark macro pass@1 climbs from 40.29% to 51% while rollout-to-learner median KL stays near zero across 20K updates; async IS without TII blows up (83.33× KL increase) around 13K updates.
- Takeaway: "REINFORCE is all you need, as long as you use the correct kernels."
More from Research
- GameVerse releases minute-scale gameplay dataset for long-horizon interactive world modeling — AlayaLab · 2026-11-05
- Solv3R: looped transformer iteratively refines multi-view geometry across reconstruction tasks — Jeong-gi Kwak · 2026-11-05
- Alibaba Qwen team's SpecOPD parallelizes on-policy distillation for faster speculative decoding — Qwen · 2026-11-05
- KAIST's H2RB transfers human demonstrations to robots, boosting object transfer from 8.0 to 51.8 — kaist-ai · 2026-11-05
- VeriSpoof: one audio LLM unifies deepfake detection, attribution and source verification — unimelb-hf · 2026-11-05
- New bioRxiv preprint measures E. coli protein yield at scale, builds predictors — sokrypton · 2026-11-05