Paper: non-LLM components dominate latency in 5 of 10 production agents
dair_ai · x · 2026-08-20
DAIR.AI highlights a paper for anyone building production-grade agents, focused on understanding what actually incurs costs in agent systems with complex components.
Key findings (across ten instrumented agentic applications):
- Non-LLM components dominate latency in five of them; within one app, task latencies diverge up to 32x across GPU-bound inference, memory-bound retrieval, and CPU-bound sandboxes.
- Idle state waste: production sessions hold state idle for minutes to hours between active steps; sandbox working sets peak at 28 GB per session.
- Control-plane tax: auxiliary LLM calls and tool schema overhead crowd out productive compute.
Optimization gains: task-aware serving cuts latency 29–40%, state offloading cuts memory 4.6x, and tool-result caching removes 35.2% of redundant search calls.
More from Infra
- LithosAI launches ultra-fast inference with Kimi K3 at 800+ tokens/s — JiaZhihao · 2026-08-20
- Baseten Hires: Inference Will Be the Largest Market Ever — philipkiely · 2026-08-20
- Deploy Qwen3.8-27B on Hugging Face for $5/hour with auto-scaling — victormustar · 2026-08-20
- Rural Residents Push Back Against Data Centers as Tech Giants Pivot Marketing — dinabass · 2026-08-20
- Startup Helps Wall Street Price AI Compute Amid Surge — TechCrunch AI · 2026-08-20
- AI uses 10x energy of a Google search, facing infrastructure supply cliff — ingliguori · 2026-08-20