NVIDIA's EPD Disaggregation Cuts Multimodal TTFT Up to 5x, E2E Latency 7x
dl_weekly · x · 2026-09-22
An NVIDIA technical blog details encode-prefill-decode (EPD) disaggregation in its Dynamo inference framework, splitting vision encoding from LLM prefill and decode stages for multimodal serving.
Key results:
- Up to 5x faster time-to-first-token and 7x faster end-to-end latency for image-heavy prompts with short-to-medium outputs
- In mixed text/multimodal traffic, removing head-of-line blocking cuts mean TTFT by 42.2% for text requests and 30.8% for image requests
- Quantizing LLM weights to NVFP4 while keeping the vision encoder in BF16 raises colocated EPD goodput from 1.78x to 2.64x over aggregated serving
Three encoder placement topologies are covered: aggregated serving, colocated encoder workers sharing GPUs with prefill-decode workers, and fully disaggregated encoders on a cheaper GPU tier connected via NIXL. Gains depend on media load, output length, model size/precision, and traffic mix, shrinking when decode dominates or large dense models reduce the encoder's compute share. Follow-ups include parallel media decoding, embedding caching, multimodal KV routing, and a reproducible guide in the ai-dynamo/dynamo repo.
More from Infra
- Gas-for-AI-datacenters forum co-located with CoRL shows energy demand spillover — carlosdponx · 2026-11-12
- a16z podcast: Mike Volpi on Cisco's 70+ acquisition playbook and the AI buildout — a16z · 2026-11-12
- AMD to pass Intel in server revenue by mid-2027 as share gains 6 points a year — BenBajarin · 2026-11-12
- Study: Reflective Data Center Roofs Actually Cool Daytime Ground by 1.8-2.2°C — SumitGup · 2026-11-12
- Tsinghua's BaGuaLu-2 tames MoE expert-load spikes at runtime, accepted to SC26 — jiqizhixin · 2026-11-12
- antirez backs low-bit DeepSeek 4.1 Flash over Qwen 27B dense on Apple M5 Macs — antirez · 2026-11-12