Paper: A Year in LLM Serving Analysis on 6.1B Requests Reveals Caching Insights
JiaZhihao · x · 2026-08-22
Based on a one-year production trace from Chutes.ai (6.1B requests, 315K users, 9,174 models), this paper analyzes the evolution of LLM serving workloads.
Key Findings:
- Workloads shift substantially over time, featuring longer inputs and shorter outputs.
- Prefix reuse is extremely temporal: 99% of repeats occur within 15 minutes.
- Simple FIFO/LRU cache policies can match or outperform more sophisticated ones.
- Caching and load balancing are tightly coupled: preserving KV locality can conflict with balancing load across replicas.
More from Infra
- Why I Wish Claude Code Harness Was Open Source — omarsar0 · 2026-08-22
- llmaker: Self-host a full modern LLM stack from your terminal — raiyanyahya · 2026-08-22
- MCP vs. RAG for local world knowledge integration? — vick2djax · 2026-08-22
- Microsoft's FinOps Control Plane Cuts AI Agent Spend by 78% — AI Engineer · 2026-08-22
- Case for an AI Shenzhen: Concentrate Clusters Where Power is Abundant — MatthewChang · 2026-08-22
- Tsinghua's PhyAI engine unifies physical AI inference across cloud and edge — jiqizhixin · 2026-08-22