From LLM Inference to Agentic Workloads: Characterization and Implications for Serving Systems
Chaokun Chang, Yukun Zhou, Kaihua Fu, Dakai An, Tianyu Feng, Hanfeng Lu, Sheng Yao, Pu Guo, Yinghao Yu, Yizhou Shan, Bo Li, Binhang Yuan, Wei Wang
cs.OS, cs.AI, cs.DC, cs.MA
2026-08-15
AgentSysBench on 10 apps and 178,799 sessions: non-LLM stages dominate 5 of 10; task-aware serving cuts latency 29-40%, state offloading cuts memory 4.6x.
Agent serving still copies chatbot assumptions: the GPU is the bottleneck, a request is one generation, and the knob to turn is tokens. Capability benchmarks such as SWE-bench and WebArena score task success and record almost none of the systems data that actually sets cost: tool latency, live state, cross-user reuse. Serving papers usually evaluate one to three apps on a frozen stack, so a reported bottleneck may belong to the hardware layout rather than the workload.
HKUST, Alibaba, and ByteDance built AgentSysBench, a systems suite of ten agent applications plus one day of production traces covering 178,799 sessions. The question is concrete: where time, memory, and money go in an agent request, and whether those facts can be turned into scheduling and caching changes.
The suite writes a workload as request mix, tools and environments, models, and orchestration, and writes a serving system as hardware, component engines, and placement. The ten apps are chosen to cover that space. RAG is a fixed pipeline. DeepResearch adds planning and parallel search. Mini-SWE, Codex, and Claude Code run ReAct loops. WebAgent and GUIAgent attach a browser and a desktop sandbox. HuggingGPT dispatches specialist models. Openclaw and Pi-AutoR cover office and research jobs that last hours.
Controlled runs use a shared Docker stack. White-box apps are annotated; black-box apps are traced through LLM/tool proxies and sandbox hooks, down to each model call, tool call, and state operation. Production adds three 24-hour traces: a coding agent (35,037 sessions), a search QA agent (141,376), and an Openclaw-like office agent (2,386). The lab side totals 4,641 requests, 64,924 LLM calls, and 118,274 tool calls.
In five of the ten apps, tools and environments dominate or co-dominate end-to-end latency. GUIAgent's desktop sandbox takes over 70%; Pi-AutoR's experiment runtime takes 90%. Requests last from seconds to hours: Mini-SWE often exceeds ten minutes, Pi-AutoR can run for hours. Sandbox working-set memory has a median peak near 0.8 GB and a per-session peak of 28 GB. A Claude Code session on DeepSeek-V4 can hold 11 GB of KV cache. Tasks on the same component still diverge: Embed-Doc is 32x slower than Embed-Query in DeepResearch; inside one Mini-SWE trace the same LLM task varies by 30x, and pip install is 171x slower than sed.
Bottlenecks also move with request type, model tier, and serving knobs. On MCP-Atlas, Claude Code spends up to 90% of time in the LLM for Movie/BI/DB tasks and up to 84% in tools for ETL/Wiki/MAP. Switching DeepResearch's writer and summarizer from DeepSeek-V4-Flash (51 tok/s) to V4-Pro (21 tok/s) raises total time from 10.0 h to 14.0 h and flips the bottleneck from embedding to the LLM. Raising GUIAgent's SGLang batch size from 1 to 4 lifts TPOT from 7 ms to 30 ms and makes the LLM the bottleneck.
Production is harsher. The median coding session spends only 20% of its lifetime executing; 70% of sessions execute for less than half their life. Idle gaps cluster between 1 and 10 minutes and can last overnight, while the per-session sandbox stays allocated. System messages start at 99.7% of the prompt and history, mostly tool observations, later reaches 84.3%. One step emits 151 tokens after prefilling 166,721. Across 3,170 compaction events, 99% cut context by more than 70%, yet one event averages 176K input tokens, 5K output tokens, and 156 s, with p99 at 775 s. A 5-minute KV-cache TTL is a poor match for human pacing: 59.4% of sessions see at least one eviction, and reprefill accounts for 31.5% of estimated spend at Claude Opus 4.6 list prices. Of 373,678 search calls, 27% of unique queries produce 67.3% of invocations.
Four design probes turn the measurements into numbers:
| Change | Baseline | Result |
| Task-disaggregated serving (Dynamic RAG) | Same GPU count, shared components | 29–40% lower latency |
| Co-locate embedding with the vector DB | One server per component | Up to 4.5x lower latency at high load; network share falls from 67.5% to near zero |
| Offload idle sandbox (Mini-SWE) | Keep sandbox resident | 4.6x less average memory, 2.1x less peak, latency up under 0.5% |
| 10-minute search-result cache | No cache | 35.2% fewer redundant searches, 19.3% less aggregate search latency |
A 10-minute URL cache also removes 11.65% of redundant fetches and 16.5% of aggregate fetch latency.
For anyone running an agent platform, making inference faster is the wrong program. Half the apps spend their critical path in sandboxes, retrieval, and browsers. Production sessions sit idle waiting for a human or the next instruction while still holding sandboxes and KV cache. A 5-minute TTL against 1–10 minute pauses burns money on reprefill. Search queries repeat across users; per-session tuning cannot see that.
The immediately usable pieces are unglamorous: query-level caches, co-locating embedding with the vector database, and splitting short query embeds from long document embeds. Task-aware serving and state offloading are directions. The paper shows proofs of concept, not a product stack.
This is a measurement paper. Its value is a map of bottlenecks, not a new engine.
The default lab setup is one GPU per workflow with containers on a shared-memory virtual network, which is not multi-tenant mix. The production traces come from three internal deployments; compaction and the 5-minute TTL are harness choices, so numbers will drift on stock Claude Code or a homegrown ReAct loop. Cost uses public-cloud list prices and Claude Opus 4.6 rates; Pi-AutoR's sandbox share above 99% depends on E2B-style per-second billing and will not match a self-hosted machine. State offloading is measured on sandbox idle time during LLM planning, not the minute-to-hour human pauses in production. The four optimizations are independent; there is no stacked end-to-end number. AgentSysBench is promised as open source; the paper does not ship a repo.