Evolving Infra beats NVIDIA Dynamo on agent workloads: 4.1x lower latency, 14x lower P95 TTFT

YouJiacheng · x · 2026-11-17

A team from MIT, Georgia Tech, UIUC, UC Berkeley, NVIDIA and OpenAI released Evolving Infra, an open-source agent-adaptive LLM serving framework with a tech report and code (mit-han-lab/EvolvingInfra).

Key ideas:

On real agent traces (computer use, coding, math agents; GLM-5.2 and Kimi K2.6) across up to 256 GB300 GPUs, it cuts mean E2E latency by up to 4.1x and P95 TTFT by up to 14x vs NVIDIA Dynamo, with up to 2.36x more revenue at list prices. Demo: 126 concurrent Claude Code agents on 36 GB300s.

Original post →

More from Infra

Infra channel →