ThunderAgent Engine: 2× Throughput, Near-Linear Multi-Node Scaling for Agent Workflows
togethercompute · x · 2026-07-30
Together Compute's ThunderAgent engine targets the inference bottlenecks in complex agent workflows. Traditional request-level engines fail to recognize that a series of LLM calls belongs to a single workflow, causing severe KV cache thrashing and cascading recomputation.
ThunderAgent tackles this by introducing a program-level scheduling abstraction:
- Smart Scheduling: Treats each agent workflow as a schedulable program, tracking its phase, KV footprint, and node placement. It pauses low-priority workflows under memory pressure to significantly boost cache hit rates.
- Single-Node Performance: On a single 8×H100 node with batch 192, compared to SGLang dropping to 390 tok/s (65s latency), ThunderAgent maintains 803 tok/s (10.6s latency)—delivering 2× throughput and roughly 6× lower latency.
- Multi-Node Scaling: Scaling from 16 to 64 GPUs increases processing speed from 671 to 2,248 steps/min, achieving near-linear scaling. The performance advantage grows with cluster size (1.79× at 2 nodes, 2.39× at 8 nodes).
- Seamless Integration: Designed as a drop-in solution compatible with existing configs like KV offloading and speculative decoding. Already adopted by SkyRL and NVIDIA Dynamo.
Related event: Together AI Unveils ThunderAgent, Doubling Agent Inference Throughput(7 posts)→
More from coding & agent
- 'LLM code is sloppy' rebutted with real ffmpeg code: identical if/else branches in a top library — kingslyj · 2026-10-16
- Agent actions need checks outside the model: NVIDIA demos lean on guardrails amid wave of real agent incidents — Creamy-And-Crowded · 2026-10-16
- Vitalik builds "paranoid mode" agent wrapper with Tor, zkAPI and local prompt rewriting — DavideCrapis · 2026-10-16
- Agent fine-tunes Qwen3.5-0.8B with synthetic data: 4.7%→76.3% accuracy, 11.9× faster — ben_burtenshaw · 2026-10-16
- Beam-501B agent demo: solo beats 5-agent team on ProgramBench rebuild task — ben_burtenshaw · 2026-10-16
- Beam agent builds BeamSpeak, a local Parakeet dictation app for macOS — ben_burtenshaw · 2026-10-16