FlashREINFORCE: critic-free single-rollout async RL stabilizes 6,000+ agentic updates
青稞AI · wechat · 2026-09-17
A technical deep-dive into FlashREINFORCE (open-sourced by the NVIDIA NeMo team), reportedly the first critic-free, single-rollout, one-pass asynchronous RL method for LLMs with 6,000+ stable reported updates.
Three components: One-Batch REINFORCE builds signed feedback across independent prompts using the batch mean reward as baseline; Sequence Trust Region screens whole trajectories via a Bernoulli KL proxy to handle async policy drift; Sample-Mean Optimization normalizes loss per trajectory so long failed trajectories don't dominate updates. No critic, ratio clipping, or reference-model forward pass needed.
Results:
- DeepSeek-R1-Distill-Qwen-1.5B long-CoT: stable at 4 policy lag for 6,000 updates, AIME24/25 mean 21.7→33.7;
- Qwen2.5-Math-1.5B: 38.0 average with 256k rollouts, +1.7 over GRPO (36.3 with 512k);
- Qwen3-30B-A3B tool use: 67.1 average at 8 policy lag, +6.8 over GRPO;
- ALFWorld: 98.3% seen / 96.5% unseen success.
Ablations confirm each design choice (36.2 vs 9.8 without negative feedback; trajectory-level admission more stable than token-level). Paper and code are open source.
More from coding & agent
- Microsoft's MemSteer turns unlabeled user histories into agent memory supervision — MicrosoftResearch · 2026-11-02
- She fired herself from her own Instagram: AI agent runs the account, views up 82.3% — heyshrutimishra · 2026-11-02
- Sourcegraph: AI writes code faster than teams can review it, driving more outages — AI Engineer · 2026-11-02
- Tsinghua & Microsoft's HarnessMM cuts agent token use by up to 87.93% while raising success rate — rohanpaul_ai · 2026-11-02
- GitHub launches Agent HQ: Anthropic, OpenAI, Google coding agents in Copilot — emmanuelvivier · 2026-11-02
- DeepSeek open-sources Super-Agent, a coding agent that reads team chats to minimize human input — emmanuelvivier · 2026-11-02