TRL's tree attention adds linear attention support with up to 3.46x speedup
SergioPaniego · x · 2026-10-27
- TRL's AsyncGRPOTrainer tree attention now supports linear attention models, with up to 3.46x speedup.
- Built on FLA's chunked Triton implementation, modifying two state-scan kernels: forward branches inherit the parent's final state; backward sums gradients from children into the parent.
- Shared prefixes are processed once, final recurrent state passed to continuations that run in parallel; gradients are summed back in backward. Causal conv1d token handling also fixed.
- Benchmarks on 1x H100 (BF16, real AsyncGRPO step): Qwen3.5-0.8B hits 3.84x on grpo-g1-64k and 4.36x on agent-fork-long; Qwen3.5-2B peaks at 4.24x, with several baseline configs OOMing where tree packing still runs.
More from Infra
- German AI experts debate their country's AI infrastructure sovereignty; a bot compiles the numbers — BlackHC · 2026-10-27
- Prisma v7 drops Rust engine for JS, doubling throughput to ~290 req/s — DanielLockyer · 2026-10-27
- Nvidia and Micron drove a third of S&P 500 profit growth last quarter — LexSokolin · 2026-10-27
- WSJ: Broadcom CEO sealed Anthropic's $63B chip debt deal with loan guarantees and higher rates — rohanpaul_ai · 2026-10-27
- Digital Sovereignty Is Now a Cloud Architecture Decision, and AI Pipelines Make It Harder — DavidLinthicum · 2026-10-27
- Houmo LQ50-24 M.2 accelerator claims 160 TOPS but real-world tests are scarce — lazy-kozak · 2026-10-27