Together AI Unveils ThunderAgent, Doubling Agent Inference Throughput
Together AI introduces ThunderAgent, a high-throughput system for large-scale agent inference, accepted as an ICML 2026 Spotlight paper. By introducing a program-level scheduling abstraction, it addresses KV cache thrashing at the scheduler level, achieving 2x single-node throughput and near-linear multi-node scaling.
Confirmed
- ThunderAgent introduces a program-level abstraction to schedule agent LLM requests. Traditional request-level engines cannot perceive correlations across calls in the same workflow, causing GPU idle time while waiting for tool calls, and traditional LRU policies blindly evict caches that will soon be reused, leading to severe KV cache thrashing and cascading recomputation.
- Performance: 2x single-node throughput; scaling from 16 to 64 GPUs improves steps/minute from 671 to 2248; 2 nodes (32 GPUs) achieve 1.79x single-node speed, 8 nodes (64 GPUs) achieve 2.39x.
- The engine is plug-and-play, requiring only a program to run.
Why it matters
Complex agent workflows often involve multiple rounds of tool calls, and traditional inference engines face severe efficiency bottlenecks in such concurrent tasks. ThunderAgent eliminates cache thrashing at the scheduling level, significantly improving inference throughput on large clusters with low integration cost, which is valuable for scenarios like multi-turn agent reinforcement learning.
2026-07-30 ~ 2026-07-30 · 7 related posts
Primary sources
- ThunderAgent (ICML 2026 Spotlight): Overcomes KV Cache Thrashing in Agentic Inference — togethercompute ·
- Together Compute unveils multi-node inference engine with near-linear scaling, adopted by SkyRL and NVIDIA Dynamo — togethercompute ·
- ThunderAgent: 2x Faster Agentic Inference for Synthetic Data Generation — togethercompute ·
- ThunderAgent by Together AI: 2x Faster Agentic Inference (ICML 2026) — togethercompute · 2026-07-30
- [source] Together Compute unveils multi-node inference engine with near-linear scaling, adopted by SkyRL and NVIDIA Dynamo — togethercompute · 2026-07-30
- [source] ThunderAgent (ICML 2026 Spotlight): Overcomes KV Cache Thrashing in Agentic Inference — togethercompute · 2026-07-30
- ThunderAgent Engine: 2× Throughput, Near-Linear Multi-Node Scaling for Agent Workflows — togethercompute · 2026-07-30
- ThunderAgent (ICML 2026): Boosts Multi-Turn Agent RL Rollout Throughput by 1.94x — simran_s_arora · 2026-07-30
2 near-duplicate retellings: togethercompute · simran_s_arora