Evolving Infra beats NVIDIA Dynamo on agent workloads: 4.1x lower latency, 14x lower P95 TTFT
YouJiacheng · x · 2026-11-17
A team from MIT, Georgia Tech, UIUC, UC Berkeley, NVIDIA and OpenAI released Evolving Infra, an open-source agent-adaptive LLM serving framework with a tech report and code (mit-han-lab/EvolvingInfra).
Key ideas:
- Retires fixed PD disaggregation: decouples compute allocation from KV cache management so nodes switch between prefill and decode in place, with KV readable by all nodes;
- Online planner: iteration-time models fitted online capture model, hardware and agent workload (including KV reuse) without offline profiling;
- KV-reuse-aware eviction and routing keeps caches agents will reuse and balances prefill work.
On real agent traces (computer use, coding, math agents; GLM-5.2 and Kimi K2.6) across up to 256 GB300 GPUs, it cuts mean E2E latency by up to 4.1x and P95 TTFT by up to 14x vs NVIDIA Dynamo, with up to 2.36x more revenue at list prices. Demo: 126 concurrent Claude Code agents on 36 GB300s.
More from Infra
- Open-source models hit 30% enterprise token share in 5 months, up from near zero — pwendell · 2026-11-17
- Databricks opens its cost-vs-quality Pareto benchmark powering the Unity Gateway Smart Router — pwendell · 2026-11-17
- Datology claims 6x compute multiplier from data curation alone; 12B beats 30B baseline — pratyushmaini · 2026-11-17
- Together AI's Five-Step Playbook for Migrating From Closed to Open Models — togethercompute · 2026-11-17
- Why tensor parallelism breaks RNG reproducibility in model init — ezyang · 2026-11-17
- Bittensor's Engy tops Venice's AI inference volume despite 28x smaller valuation — markjeffrey · 2026-11-17