Decoding NVIDIA Dynamo's router: Unifying GPU load and cache into one metric
Abhishekcur · x · 2026-07-28
The author provides a deep dive into NVIDIA Dynamo's LLM routing mechanism. Its core design converts waiting work, occupied space, and skippable cache into the same unit (a block of cache), reducing each GPU's state to a single number. The router simply picks the GPU with the smallest number.
Trade-offs between Cache and Space
- Memory as a Discount: Remembering a user's prompt doesn't dictate routing direction but acts as a cost discount. The discount depends on where the memory resides: highest if on the GPU, lower in normal memory, and lowest on disk.
- Space vs. Work: Space refers to the actual resources consumed while a request runs, while memory (cache) merely saves you from redundant reading work. The two are fundamentally distinct.
Queue Time Determines the Winner
- The primary factor the router evaluates is the GPU's current backlog of reading work (queue length), rather than just the size of the new request.
- Tests show that even if a GPU has perfect prompt memory, if it is busy (long queue), it will still be easily beaten by a free GPU with no memory.
Related event: How NVIDIA Dynamo Prices LLM GPU Routing Instead of Hard KV Rules(14 posts)→
More from Infra
- Dedicated Hardware for Running AI Agents at Scale Arrives — cyrilzakka · 2026-09-23
- Ternary Bonsai 2 27B: 5.9GB weights retain ~95% of full-precision reasoning — cephaloform · 2026-09-23
- Qwen 27B runs 24hr unattended on one RTX5090, builds full Postgres-SpringBoot-React spreadsheet app — anglepoiselife · 2026-09-23
- OpenRoboto Shift launches: decentralized egocentric video data network for robot brains — markjeffrey · 2026-09-23
- Engineer describes designing digital circuits that recycle most of their energy — MikePFrank · 2026-09-23
- Cloudflare CTO Dane Knecht makes TIME's 2026 executives list as AI crawlers hit 52% of traffic — dinasaur_404 · 2026-09-23