Paper: A Year in LLM Serving Analysis on 6.1B Requests Reveals Caching Insights
JiaZhihao · x · 2026-08-22
Based on a one-year production trace from Chutes.ai (6.1B requests, 315K users, 9,174 models), this paper analyzes the evolution of LLM serving workloads.
Key Findings:
- Workloads shift substantially over time, featuring longer inputs and shorter outputs.
- Prefix reuse is extremely temporal: 99% of repeats occur within 15 minutes.
- Simple FIFO/LRU cache policies can match or outperform more sophisticated ones.
- Caching and load balancing are tightly coupled: preserving KV locality can conflict with balancing load across replicas.
More from Infra
- NVIDIA releases Nemotron 3.5 Bolt 74B MoE that fits in 24GB VRAM at 2-bit — NerdyRodent · 2026-10-16
- Analyst: 80% of neoclouds may not survive, but the demand for their GPU capacity is real — BenBajarin · 2026-10-16
- Apple and Meta partner to run Muse Spark model on linked Mac Studios — pdamodaran · 2026-10-16
- Nvidia's developer base hits 11 million, up ~3.7x from 3 million in 2021 — Beth_Kindig · 2026-10-16
- Muse Spark Open runs locally on a Mac M3 Ultra via llama.cpp: frontier model on desktop — pcuenq · 2026-10-16
- Muse-Spark open-weights drop: 816B-A42B MoE model gets 359GB MLX 3-bit build — ns123abc · 2026-10-16