Tree packing lands in TRL's AsyncGRPO, cutting agentic RL training compute by over 26x
joecole · x · 2026-10-16
The TRL team shipped tree packing in AsyncGRPO, reducing trainer compute by over 26x to reach the same reward on agentic RL.
- Problem: async RL works best with large batches, but long rollouts (>100k tokens in agentic training) make the fwd/bwd pass the bottleneck. GRPO forks the same prompt G times and multi-turn rollouts share most of their history — sequence packing forwards shared tokens once per sample.
- Approach: tree packing uses a tree data structure to share prefixes across trajectories, forwarding each unique token exactly once while each token keeps its own target, advantage, and previous log-prob. Shipped implementation is a single FA3 varlen call (v1 used FlexAttention).
- Validation: extensively tested against sequence packing on dense, sliding-window, softcap, MoE, and VLM models; 1% overhead when nothing packs. On a real agentic RL run on MiMo Envs it reached the same reward with 26x less compute.
Related event: TRL Adds Tree Packing, Cutting Agentic RL Compute Over 26x(2 posts)→
More from Infra
- Local AI is a routing problem, not a privacy feature: where each task should run — bigdata · 2026-10-16
- Moving agent work from GPUs to CPUs for durable, predictable actions — cramforce · 2026-10-16
- First TPUv7 agentic benchmark on vLLM claims 1.42x perf-per-dollar over GB300 NVL72 — vllm_project · 2026-10-16
- Phison CEO calls CXMT memory scare a 'ghost story', warns of materials shortage — BenBajarin · 2026-10-16
- Ben Bajarin: The Future of Compute Is Fungible as Hyperscalers Design for Workload Flexibility — BenBajarin · 2026-10-16
- AirLLM runs frontier-size models layer-by-layer: 671B on 12GB VRAM — Aiden_Tech_Ai · 2026-10-16