ThunderAgent: A Simple, Fast and Program-Aware Agentic Inference System
Hao Kang, Ziyang Li, Weili Xu, Xinyu Yang, Yinfang Chen, Junxiong Wang, Beidi Chen, Tushar Krishna, Chenfeng Xu, Simran Arora
cs.OS, cs.MA
2026-02-14
ThunderAgent treats an entire agent workflow as one schedulable program, keeping KV cache hot during tool calls. Serving throughput hits 1.5-3.6x over vLLM, RL rollout up to 3.9x.
Running a tool-using agent, say a model that writes code and compiles it in a sandbox, means bouncing between LLM inference and tool execution dozens of times. Today's deployments glue together two separate systems: an inference engine like vLLM and a tool orchestrator like Kubernetes. Each allocates resources per request, and neither knows what the whole workflow looks like.
This breaks in three ways. Under high concurrency, the engine frees GPU memory by evicting the KV cache of workflows waiting on tools; when the tool returns, it must recompute the entire context, stretching a single request's latency up to 7.14x. To preserve cache hits, routers pin one workflow to one node, which unbalances memory across nodes: over a 90-minute rollout the gap hits 51%, and imbalance above 20% persists for 37 minutes. Tool sandboxes are never cleaned up, so disk and network ports leak away.
The root cause is request-level scheduling with no end-to-end view of the workflow.
ThunderAgent abstracts an agent workflow as a "program," tracking its context length (how much KV cache it occupies), the set of tool environments it uses, which node it sits on, whether it is reasoning or calling a tool, and its scheduling state (active, paused, terminated). With this one abstraction, resources that were managed separately, like GPU memory, system state, and external disk and network ports, can be scheduled together.
The scheduler is built around one goal: keep the KV cache hit rate high. Every few seconds (default 5s) it samples each node's memory; when a node is about to thrash, it pauses the programs with the shortest context first, because the cost of recomputing a workflow's KV cache scales quadratically with its length, so evicting short ones is cheapest. For programs currently calling tools (which do not need their KV right now), a time-decay function of 2 to the minus t progressively lowers their memory weight: the longer a tool drags on, the more willing the scheduler is to give up its cache. The paper proves that when tool latency is memoryless (remaining time independent of elapsed time), exponential decay is the only optimal form.
On the tool side, lifecycle hooks tear down sandboxes and API servers the moment a program terminates, plugging leaks; when a high-priority program is about to resume, its environment is prepared asynchronously so the initialization I/O hides behind inference. Finally, all nodes share a global waiting queue. A paused program does not have to return to its original node; it routes to whichever node has free capacity, putting idle cross-node memory to work.
Across three workload types, coding (SWE-Bench), routing (HLE-Bench), and scientific discovery (ScienceAgentBench), with GLM-4.6 (355B) and Qwen-3 (235B) in FP8 on 8xH100 nodes:
| Workflow | Model | Speedup over vLLM |
| OpenHands coding | GLM-4.6 | 3.58x |
| mini-SWEAgent | Qwen-3 235B | 3.02x |
| ScienceAgent discovery | GLM-4.6 | 1.24x |
| ToolOrchestra routing (HLE) | Qwen3-8B | 1.48x |
Overall serving throughput is 1.48 to 3.58x over vLLM and 1.17 to 3.31x over Continuum, the dedicated multi-turn system. For workflows with deterministic tool times, ThunderAgent keeps the KV hit rate near 100%, while Continuum drops from over 90% to about 60% under load. On RL rollout (two 8xH100 nodes), mini-SWEAgent goes from 375 to 672 steps/min (1.79x) and OpenHands from 69 to 271 steps/min (3.92x). Tool resource management saves 4.2x disk. The system is integrated into SkyRL and NVIDIA Dynamo, scales to 64 H100s, and also works on A100.
Inference cost is shifting from single Q&A turns to running whole agent workflows, and most of that cost is waiting on tools and recomputing evicted context. The contribution is a scheduling layer that treats agent inference as structured programs rather than isolated requests, and it is engine-agnostic: it wraps vLLM or SGLang and needs only three changes to adopt. For teams building agent training or serving, this is a throughput win they can use directly. The caveat is that the gains come from scheduling under high concurrency; a single low-concurrency run will not change much.
Workflows with highly random tool times, such as ToolOrchestra calling external model APIs, are the weak spot. In one configuration ThunderAgent is actually 0.65x the baseline, and the paper concedes it trades KV hit rate for compute utilization there. Every evaluation uses GLM-4.6 or Qwen-3; other model families are untested. Multimodal agents, consumer-grade GPUs (besides one RTX 5090), and fault tolerance at scale are not covered. Scheduler overhead grows from 13.5ms at 2 nodes to 22.7ms at 8 nodes, acceptable so far, but no data exists for larger clusters.