Harvard study: LLM serving efficiency relies on traffic patterns
rohanpaul_ai · x · 2026-08-18
A joint study by Harvard and UChicago analyzed 6.12B requests across 9,174 models over one year of production data.
Key Findings:
- High Reusability: 99% of context reuse happens within 15 minutes, as users often return to the same model with growing contexts.
- Scheduling Strategy: Simple load balancing can hurt KV Cache reuse advantages; request history should be considered.
- Dynamic Traffic: Model popularity shifts monthly, new users send longer inputs, and outputs tend to shorten over time.
Conclusion: Synthetic benchmarks fail to reflect long-term production environments. Routing, caching, and capacity planning should leverage the history and structure of real traffic.
More from Infra
- HBM Inventor Criticizes Samsung's zHBM: Stacking Memory on GPU Causes Melting — zephyr_z9 · 2026-08-18
- Five DGX Sparks Run Hot Serving vLLM — And It's the CPU, Not the GPU — EAccelerate_42 · 2026-08-18
- Ultra-fast Fourier transform and optical AI with a single lens — MeasurementDull7350 · 2026-08-18
- Dashboard Tracks OpenRouter Inference Provider Market Share and Revenue — Ronangmi · 2026-08-18
- Building distributed training framework from scratch — hkproj · 2026-08-18
- Rebellions Exec on Sovereign AI: Infrastructure, Data, and Expertise — DavidBennett__ · 2026-08-18