Paper: A Year in LLM Serving Analysis on 6.1B Requests Reveals Caching Insights
JiaZhihao · x · 2026-08-22
Based on a one-year production trace from Chutes.ai (6.1B requests, 315K users, 9,174 models), this paper analyzes the evolution of LLM serving workloads.
Key Findings:
- Workloads shift substantially over time, featuring longer inputs and shorter outputs.
- Prefix reuse is extremely temporal: 99% of repeats occur within 15 minutes.
- Simple FIFO/LRU cache policies can match or outperform more sophisticated ones.
- Caching and load balancing are tightly coupled: preserving KV locality can conflict with balancing load across replicas.
More from Infra
- Copper to optical: optical backplanes seen as necessary as racks move to optics — BenBajarin · 2026-10-16
- Polymarket prices 28% odds a US state enacts data center moratorium by year-end — Polymarket · 2026-10-16
- Pennsylvania residents reject $10,000-per-household offer for massive Amazon data center — Polymarket · 2026-10-16
- PyTorch now runs natively on AWS Trainium with zero code changes — PyTorch · 2026-10-16
- Chinese labs run on ~1/10 US compute; ByteDance pays 2-4x to hoover up SEA supply — robleclerc · 2026-10-16
- Zymtrace raises $12.2M and hires ML engineers to build agents that fix GPU bottlenecks — cfregly · 2026-10-16