GLM Team Proposes SAO Algorithm for Asynchronous Agent RL

Tsinghua and Z.AI proposed SAO (Single-rollout Asynchronous Optimization) for LLM agent reinforcement learning. This method allows the model to learn immediately after a single rollout completes, breaking the limitation of waiting for the slowest trajectory in group-wise sampling and addressing off-policy bias in long-sequence asynchronous training.

Core Mechanism and Training Details

The core of SAO involves shifting from group-wise to single-rollout sampling. To reduce the resulting variance, a value model is reintroduced. For agent trajectories, to prevent the advantage function from passing through ungenerated observation tokens, the last token of an action is directly connected to the next action. Regarding Critic training, the team froze attention layers and only updated MoE projections to mitigate gradient explosions.

Off-policy Handling and Performance

To handle off-policy bias, the algorithm directly uses the rollout policy for importance sampling and simplifies trust region clipping. Specifically, it discards the old policy, uses the logprob ratio of the current policy and rollout engine, and applies bilateral hard boundaries to drop out-of-bounds samples directly in the gradient. Experiments show that GRPO degrades severely in asynchronous settings, whereas SAO remains stable. SAO outperforms existing methods on coding and reasoning benchmarks like SWE-Bench Verified, and adapts faster in online experiments with shifting preferences using a GLM 4.7 judge.

2026-07-09 ~ 2026-07-11 · 15 related posts

• Single-Rollout Async Optimization for Agent RL — the-aiml · 2026-07-09
• [source] Single-Expansion Async Optimization Boosts Agent RL — zai-org · 2026-07-09
• [source] Notes on the GLM 5.2 RL Paper — nrehiew_ · 2026-07-09
• Analyzing GLM Algorithm Off-Policy Handling — nrehiew_ · 2026-07-09
• Details of Long-Sequence RL Training — nrehiew_ · 2026-07-09
• SAO Adapts Faster in Online Experiments — nrehiew_ · 2026-07-09
• SAO Details in GLM 5.2 RL — nrehiew_ · 2026-07-09
• Asynchronous Agent Training via Single Rollout — burny_tech · 2026-07-10
• [source] Tsinghua/Z.AI Publish Paper on Asynchronous RL for Agents — ziv_ravid · 2026-07-11
• Asynchronous Reinforcement Learning Unsuitable for GRPO — ziv_ravid · 2026-07-11
• Critic Training Details and Stability — ziv_ravid · 2026-07-11
• GAE Processing for Agent Trajectories — ziv_ravid · 2026-07-11
• Handling Hard Boundaries in Off-Policy Training — ziv_ravid · 2026-07-11
• SAO Outperforms GRPO Variants — ziv_ravid · 2026-07-11
• Study Confirms Mismatch in Async and Group Sampling — ziv_ravid · 2026-07-11