Agentic traffic now exceeds 70% of all inference traffic, says SemiAnalysis
NinaDSchick · x · 2026-09-18
SemiAnalysis reports that agentic traffic now makes up over 70% of all inference traffic. Agentic workloads have four defining traits:
- Multi-turn: tens to hundreds of turns per session, enabling heavy KV-cache reuse
- Long context: system prompts, tool definitions, and turn count make context grow fast
- High prefix reuse: linear conversation means turn n's input embeds turn n-1's output, so cached-vs-recomputed input ratio approaches 1 as n grows (storage permitting)
- Sub-agent bursts: sessions spawn short-lived sub-agents with fresh context, creating bursty KV-cache patterns
Big implications for KV-cache storage and serving architectures.
More from AGI Musings
- 'AI is a massive balance patch and elo reset in the game of life—and everyone's still playing the old meta' — zack_overflow · 2026-11-02
- Why AI isn't 'alien intelligence': a tree is stranger than any model — LucaAmb · 2026-11-02
- Mathematician Tony Feng: current-paradigm LLMs lack the diversity of thought research needs — littmath · 2026-11-02
- METR's independent probe of the OpenAI agents' multi-day Hugging Face hack — gleech · 2026-11-02
- MIT Tech Review: The agentic web is here — AI agents are remaking the internet — MIT Tech Review AI · 2026-11-02
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02