Harvard lab releases one year of LLM inference traces: 6.12B production requests across 9,174 models
airesearch12 · x · 2026-09-20
Harvard SEAS's MadSys Lab, with Chutes and FreeInference, has open-sourced a year of production LLM inference metadata: 6.12 billion requests spanning 9,174 models, with request arrivals, input/output token counts, cache hits, and TTFT, plus a paper and code for serving-workload research.
Released datasets:
- A Year in LLM Serving: a year of Chutes production traces covering model usage, tokens, and prefix-cache reuse (userid rotates every three months)
- GPU and serving telemetry: time-series from GPUs and inference engines
- Coming soon: Chutes conversational/roleplay dataset, agent-serving data (block hashes, timing, tool calls), and raw prompts
Early findings:
- Strong temporal locality: 99% of repeat requests from the same user to the same model arrive within 15 minutes
- Simple LRU often matches or beats more complex cache eviction policies
- Cache-aware routing raises token hit ratio, with 5–7% load imbalance in simulation
Related event: Harvard Releases Dataset of 6.12 Billion Real-World LLM Inference Requests(3 posts)→
More from Infra
- Agent Substrate Calculator Estimates Session Resume Speed Under Load — pjausovec · 2026-11-12
- Dell expands AI Data Platform: agents have a data-context problem, not a model problem — DavidLinthicum · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- Potion UI's local login system turns your PC into a multi-user AI server — cocktailpeanut · 2026-11-12
- AWS Finally Built Its Own AWS Emulator — super_linear · 2026-11-12
- Megakernel is multiprogramming for GPUs: 'GPUs are entering the CPU age' — pranjalssh · 2026-11-12