REINFORCE Zero: TII plus minimal IS ratio keeps async LLM RL stable for 20,000 updates, open-sourced
kastnerkyle · x · 2026-11-06
Open-source REINFORCE Zero combines Trainer–Inference Invariance (TII) with the simplest importance sampling ratio to keep async LLM RL stable for 20,000 policy updates — claimed as the first open 20K demo of this recipe: unclipped token IS, no rejection gates, minimal loss.
- Macro pass@1 across five benchmarks climbs from 40.29% to 51%, matching sync TII; median rollout-to-learner KL stays flat while standard async IS blows up 83.33x
- Gradient norms stay controlled (0.05 peak vs 38.27 for async IS) with 0% sequence rejection
- Also shows lag-64 training with Qwen3-4B and Python tools over 5K updates, reaching 45% average pass@1 on AIME24/25/26
Related event: REINFORCE Zero: Open-Source Asynchronous LLM RL Runs Stably for 20K Steps(3 posts)→
More from coding & agent
- FastAPI creator tiangolo says he now uses voice-driven Codex way more after jxnlco's talk — jxnlco · 2026-11-06
- AWS WAF + Stripe lets sites charge AI agents via HTTP 402, with fiat payouts — jeff_weinstein · 2026-11-06
- The computable enterprise: capturing business logic that lives in employees' heads — vishalmisra · 2026-11-06
- Claude skills are useless — until you hook them to your actual business — aryanXmahajan · 2026-11-06
- Do you use MCP firewalls with Claude, Codex or Cursor in production? — vdobroskok · 2026-11-06
- Firecrawl launches Alexandria agent data layer with Benzinga, announces $75M Series B — devdigest · 2026-11-06