Agentic traffic now exceeds 70% of all inference traffic, says SemiAnalysis
NinaDSchick · x · 2026-09-18
SemiAnalysis reports that agentic traffic now makes up over 70% of all inference traffic. Agentic workloads have four defining traits:
- Multi-turn: tens to hundreds of turns per session, enabling heavy KV-cache reuse
- Long context: system prompts, tool definitions, and turn count make context grow fast
- High prefix reuse: linear conversation means turn n's input embeds turn n-1's output, so cached-vs-recomputed input ratio approaches 1 as n grows (storage permitting)
- Sub-agent bursts: sessions spawn short-lived sub-agents with fresh context, creating bursty KV-cache patterns
Big implications for KV-cache storage and serving architectures.
More from AGI Musings
- Noam Brown: models may perform their chain of thought; alignment must be solved — infoxiao · 2026-09-18
- Redditor argues AI hype is manufactured by AI companies as LLMs stall — EddieMidz · 2026-09-18
- The 'Seniority Cliff': skipping junior-level friction may hollow out engineering intuition — Jumpy-Increase9337 · 2026-09-18
- e/acc Manifesto: Doomers Fear the Future, Builders Create It — mark_k · 2026-09-18
- DeepMind vet vqctran leaves after 10 years to co-found biology AI startup Polyphron — infoxiao · 2026-09-18
- "Looping on an LLM core" agent architecture is deeply flawed, argues dev — GaryMarcus · 2026-09-18