Evolving Infra: agent-adaptive LLM serving cuts latency up to 4.1x vs NVIDIA Dynamo
mit-han-lab · hf · 2026-11-12
MIT Han Lab, with Georgia Tech, UIUC, UC Berkeley, NVIDIA, OpenAI and others, released Evolving Infra, an open-source agent-adaptive LLM serving framework.
- Problem: agentic workloads dominate LLM inference demand, but existing prefill–decode (PD) disaggregated serving systems can't adapt compute allocation at runtime and evict KV caches agents will reuse, causing load imbalance.
- Approach: decouple compute allocation from KV cache management so nodes switch efficiently between prefill and decode; an online planner matches the PD split to workload composition; an agent-aware distributed KV manager uses agent context-management signals to guide cache eviction and transfer.
- Results: on coding, math and computer-use tasks with GLM-5.2 and Kimi K2.6 on up to 256 GB300 GPUs, Evolving Infra reduces mean end-to-end latency by up to 4.1x and P95 TTFT by up to 14x versus NVIDIA's production framework Dynamo (Claude Code, 90 concurrent agents).
- Open source: github.com/mit-han-lab/EvolvingInfra
More from Infra
- Nebius Q3: $12B in customer commitments, revenue up 68% to $981M as GPU prices climb 22% — demian_ai · 2026-11-12
- 1T-Parameter MoE trained on EU public supercomputer with PyTorch, per-GPU throughput up 14x — PyTorch · 2026-11-12
- llama.cpp crosses 134k GitHub stars as the default local LLM inference stack — ggml-org · 2026-11-12
- TensorFold: a Zig LLM inference engine spanning Metal, CUDA and Vulkan — ashhart · 2026-11-12
- TinyTapeout Explains How Open-Source Chips Are Factory-Tested Before Shipping — Obijuan_cube · 2026-11-12
- Tesla deployed 46.7 GWh of battery storage in 2025, over 15% of the global total — XFreeze · 2026-11-12