Harvard lab releases one year of LLM inference traces: 6.12B production requests across 9,174 models
airesearch12 · x · 2026-09-20
Harvard SEAS's MadSys Lab, with Chutes and FreeInference, has open-sourced a year of production LLM inference metadata: 6.12 billion requests spanning 9,174 models, with request arrivals, input/output token counts, cache hits, and TTFT, plus a paper and code for serving-workload research.
Released datasets:
- A Year in LLM Serving: a year of Chutes production traces covering model usage, tokens, and prefix-cache reuse (userid rotates every three months)
- GPU and serving telemetry: time-series from GPUs and inference engines
- Coming soon: Chutes conversational/roleplay dataset, agent-serving data (block hashes, timing, tool calls), and raw prompts
Early findings:
- Strong temporal locality: 99% of repeat requests from the same user to the same model arrive within 15 minutes
- Simple LRU often matches or beats more complex cache eviction policies
- Cache-aware routing raises token hit ratio, with 5–7% load imbalance in simulation
More from Infra
- Large Power Transformer Lead Times Hit 2029, Prices Up 77% Since 2019, Choking AI Data Center Buildout — sahilypatel · 2026-09-20
- Athena engine runs DeepSeek V4 + Qwen3.8 at 262K context on one DGX Spark — solyarisoftware · 2026-09-20
- Dev benchmarks Bend 2 on M4 Max: parallel kernel 6.4x faster than NumPy — arthurcolle · 2026-09-20
- China's CXMT says new memory-chip platform enters mass production — johnnyApplePRNG · 2026-09-20
- Unverified claim: OpenAI is out of compute — ns123abc · 2026-09-20
- awesome-local-ai: one-command local AI stacks with coding agents, benchmarked on real hardware — julianharris · 2026-09-20