FlashREINFORCE: critic-free single-rollout async RL stabilizes 6,000+ agentic updates
青稞AI · wechat · 2026-09-17
A technical deep-dive into FlashREINFORCE (open-sourced by the NVIDIA NeMo team), reportedly the first critic-free, single-rollout, one-pass asynchronous RL method for LLMs with 6,000+ stable reported updates.
Three components: One-Batch REINFORCE builds signed feedback across independent prompts using the batch mean reward as baseline; Sequence Trust Region screens whole trajectories via a Bernoulli KL proxy to handle async policy drift; Sample-Mean Optimization normalizes loss per trajectory so long failed trajectories don't dominate updates. No critic, ratio clipping, or reference-model forward pass needed.
Results:
- DeepSeek-R1-Distill-Qwen-1.5B long-CoT: stable at 4 policy lag for 6,000 updates, AIME24/25 mean 21.7→33.7;
- Qwen2.5-Math-1.5B: 38.0 average with 256k rollouts, +1.7 over GRPO (36.3 with 512k);
- Qwen3-30B-A3B tool use: 67.1 average at 8 policy lag, +6.8 over GRPO;
- ALFWorld: 98.3% seen / 96.5% unseen success.
Ablations confirm each design choice (36.2 vs 9.8 without negative feedback; trajectory-level admission more stable than token-level). Paper and code are open source.
More from coding & agent
- Diorama gives OpenAI Codex coding agents a visual office you can watch work in real time — davidfromkansas · 2026-09-17
- Code-first, UI on top: building bespoke brand design tools with AI — floguo · 2026-09-17
- Study of 7 models across Claude Code, Codex, Pi: harness barely affects success but swings cost — DavideCrapis · 2026-09-17
- AI trading bot built with Jev is down 85%, owner shrugs it off — generativist · 2026-09-17
- Redditor's 3-Day SoL-Pi Test: Memory Objects Save ~12k Tokens Per Tool Run — Garblyx · 2026-09-17
- Reviewing AI code through Steve Jobs' lens: unseen internals deserve beauty too — sergeykarayev · 2026-09-17