NVIDIA's EPD Disaggregation Cuts Multimodal TTFT Up to 5x, E2E Latency 7x
dl_weekly · x · 2026-09-22
An NVIDIA technical blog details encode-prefill-decode (EPD) disaggregation in its Dynamo inference framework, splitting vision encoding from LLM prefill and decode stages for multimodal serving.
Key results:
- Up to 5x faster time-to-first-token and 7x faster end-to-end latency for image-heavy prompts with short-to-medium outputs
- In mixed text/multimodal traffic, removing head-of-line blocking cuts mean TTFT by 42.2% for text requests and 30.8% for image requests
- Quantizing LLM weights to NVFP4 while keeping the vision encoder in BF16 raises colocated EPD goodput from 1.78x to 2.64x over aggregated serving
Three encoder placement topologies are covered: aggregated serving, colocated encoder workers sharing GPUs with prefill-decode workers, and fully disaggregated encoders on a cheaper GPU tier connected via NIXL. Gains depend on media load, output length, model size/precision, and traffic mix, shrinking when decode dominates or large dense models reduce the encoder's compute share. Follow-ups include parallel media decoding, embedding caching, multimodal KV routing, and a reproducible guide in the ai-dynamo/dynamo repo.
More from Infra
- Full-Parameter RL on TPUs: peano_ai Runs 310B MiMo-V2.6 Across 1,000+ TPUs — simonguozirui · 2026-09-22
- Fighting AI crawler traffic: beyond Turnstile, Cloudflare's AI Labyrinth as an option — fforres · 2026-09-22
- Software moats won't survive RSI — ML infra's value is demand aggregation, says cHHillee — PatrickToulme · 2026-09-22
- Raspberry Pi locks devices to original RAM size, blocking aftermarket memory upgrades — ngxson · 2026-09-22
- fal's H3 Max generates 5 seconds of frontier-quality video in just 3 seconds — gorkem · 2026-09-22
- Running MiniMax H3 locally on a 16GB Mac: 8-10s clips in 15-20 minutes — coberholzer · 2026-09-22