ThunderAgent (ICML 2026 Spotlight): Overcomes KV Cache Thrashing in Agentic Inference
togethercompute · x · 2026-07-30
Agent workflows alternate between GPU-heavy reasoning and GPU-idle waiting on tools. Running hundreds concurrently causes KV cache thrashing, where dumb LRU policies blindly evict caches needed seconds later, leading to cascading recomputations.
ThunderAgent fixes this at the scheduler level. It achieves 2.5x higher single-node throughput and 10x lower P50 latency at high concurrency. The paper was accepted as an ICML 2026 Spotlight.
Related event: Together AI Unveils ThunderAgent, Doubling Agent Inference Throughput(7 posts)→
More from coding & agent
- Connected steps are not a workflow: nobody designs the handoff — usehive · 2026-10-16
- Making Blender animations with Opus, Astra and Argon: how you prompt still matters — chongdashu · 2026-10-16
- Running PolyStrata + Cline coding agent on a $0-fancy laptop at ~90 tok/s — rpwoerk · 2026-10-16
- How do you attribute agent gains: model training or infrastructure? — jrxna · 2026-10-16
- Feed Claude your books, blogs and top engineers' code for claude.md — 'it now writes better code than me' — blaizedsouza · 2026-10-16
- Open-source /label plugin auto-names and colors Claude Code sessions in one step — hermanschutte · 2026-10-16