Together AI Unveils ThunderAgent for 2x Agent Inference Speedup
Together AI has introduced ThunderAgent, a high-throughput system designed for large-scale agent inference. Accepted as a Spotlight paper at ICML 2026, the system introduces a novel program-level scheduling abstraction to resolve performance bottlenecks in agent workflows, achieving near-linear scaling across multiple nodes.
已确认
- 要点 ThunderAgent introduces a program-level abstraction to schedule LLM requests for agents. Traditional request-level engines are unaware of the dependencies between multiple calls within the same workflow, leading to GPU idle times while waiting for tool calls. Furthermore, conventional LRU policies blindly evict caches that are about to be reused, causing severe KV cache thrashing and cascading recomputations. The new system addresses these issues directly at the scheduler level.
- 要点 In terms of performance, ThunderAgent achieves a 2x increase in single-node throughput.
- 要点 This multi-node inference engine supports near-linear scaling and is plug-and-play, requiring just a single program to run. Scaling tests show that when expanding from 16 GPUs to 64 GPUs, steps per minute increase from 671 to 2248. Its performance lead grows with cluster size: 2 nodes (32 GPUs) are 1.79x faster than a single node, and 8 nodes (64 GPUs) are 2.39x faster.
为什么重要
- 要点 Complex agent workflows typically involve multi-turn tool calls, where traditional inference engines face severe efficiency bottlenecks when handling such concurrent tasks. By tackling the underlying scheduling mechanism to eliminate cache thrashing, ThunderAgent significantly boosts inference throughput across large-scale clusters with minimal integration overhead.
2026-07-30 ~ 2026-07-30 · 6 related posts
Primary sources
- ThunderAgent (ICML 2026 Spotlight): Overcomes KV Cache Thrashing in Agentic Inference — togethercompute ·
- Together Compute unveils multi-node inference engine with near-linear scaling, adopted by SkyRL and NVIDIA Dynamo — togethercompute ·
- ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation — togethercompute ·
- ThunderAgent by Together AI: 2x Faster Agentic Inference (ICML 2026) — togethercompute · 2026-07-30
- [source] Together Compute unveils multi-node inference engine with near-linear scaling, adopted by SkyRL and NVIDIA Dynamo — togethercompute · 2026-07-30
- [source] ThunderAgent (ICML 2026 Spotlight): Overcomes KV Cache Thrashing in Agentic Inference — togethercompute · 2026-07-30
- ThunderAgent Engine: 2× Throughput, Near-Linear Multi-Node Scaling for Agent Workflows — togethercompute · 2026-07-30
- ThunderAgent (ICML 2026): Boosts Multi-Turn Agent RL Rollout Throughput by 1.94x — simran_s_arora · 2026-07-30
1 near-duplicate retellings: togethercompute