Zhipu's SAO: single-rollout async RL trains stably for 1,000 steps, beats GRPO
teortaxesTex · x · 2026-08-21
Zhipu's paper SAO (Single-Rollout Asynchronous Optimization) tackles off-policy and stability challenges in asynchronous RL for agentic tasks:
- Replaces GRPO-style group sampling with single-rollout sampling (one rollout per prompt)
- Adds practical value-model training designs
- Introduces a strict two-sided token-level clipping strategy for stability
SAO trains stably for 1,000 steps and consistently outperforms GRPO variants on agentic benchmarks like SWE-Bench Verified, BeyondAIME and IMOAnswerBench. It is already deployed in the agentic RL pipeline for GLM-5.2.
Related event: Zhipu's SAO: Stable Asynchronous RL for Long-Horizon Agents(2 posts)→
More from Models
- Muse Spark Open runs locally on a Mac M3 Ultra via llama.cpp: frontier model on desktop — pcuenq · 2026-10-16
- OpenAI ships GPT-Live-1.1: new voices in six languages, fewer confirmations — OpenAI · 2026-10-16
- Muse Spark model appears on LMArena, testable in Battle and Agent modes — arena · 2026-10-16
- Running a multi-model UI design arena: Paseo orchestrates four design drafts per model — vista8 · 2026-10-16
- Ilya Sutskever questions Meta license clause taxing MaaS providers above $20M revenue — ilyasu · 2026-10-16
- Muse-Spark open-weights drop: 816B-A42B MoE model gets 359GB MLX 3-bit build — ns123abc · 2026-10-16