How NVIDIA Dynamo Prices LLM GPU Routing Instead of Hard KV Rules

NVIDIA Dynamo’s LLM router is being analyzed by multiple technical authors as a pricing system rather than a hard-rule KV router. The key idea is to convert queued work, runtime occupancy, and cache reuse into the same unit—cache blocks—so each GPU gets a single total cost and the router picks the lowest one. That matters because prompt locality alone can overload the GPU with the “best memory,” while Dynamo’s queue-aware discounting lets cache advantages fade as that GPU gets busier.

Confirmed

• According to breakdowns by @Abhishekcur and @toddhooper, the routing formula’s first term is not just the new request’s size. It estimates the GPU’s unfinished read work: work already accepted but not completed, plus the read demand introduced by the new request.
• A separate term models space/occupancy. @Abhishekcur notes this is based on resources already consumed by running requests plus the extra footprint the new request would add; it is explicitly different from whether a prompt is already cached.
• Dynamo then assigns prices to reuse at different cache levels. Reuse from GPU cache gets the largest discount, reuse from system memory gets a smaller one, and reuse from disk gets the smallest.
• These dimensions are all converted into the same unit, cache blocks, compressing each GPU’s state into a single score. Routing becomes a simple “send to the lowest-score GPU” decision rather than a multi-stage rule chain.
• @Abhishekcur also highlights that the cache discount is relative to the currently emptiest eligible GPU, not calculated from zero, and that queue length is measured in terms of the request’s own size rather than a fixed threshold.

Why it matters

• Several authors argue that naive prompt/KV routing has a classic failure mode: if requests are always sent to the GPU that already remembers the prompt, that GPU can become increasingly overloaded while others sit idle.
• Dynamo’s “price, not rule” design avoids turning cache hits into an absolute priority. As a GPU’s queue grows, the benefit of cached memory reuse shrinks relative to a less busy GPU, enabling global load balancing without discarding cache locality entirely.
• For multi-GPU LLM serving, this is a useful framing: memory reuse is treated as an economic advantage that competes against queueing and occupancy, not as a separate hard constraint.

2026-07-27 ~ 2026-07-28 · 14 related posts

• [source] Dynamo scores GPUs by outstanding read work, not request size — toddhooper · 2026-07-27
• Dynamo shows why prompt memory alone can overload one GPU — Abhishekcur · 2026-07-27
• Dynamo turns routing into one cache-block cost per GPU — Abhishekcur · 2026-07-27
• Dynamo prices GPU, RAM, and disk cache reuse differently — Abhishekcur · 2026-07-27
• Dynamo separates occupied space from cached memory in routing — Abhishekcur · 2026-07-27
• Dynamo routes by total cache-block cost, not by prompt memory alone — Abhishekcur · 2026-07-27
• GPU routing uses queue-aware memory discounts instead of a fixed reuse rule — Abhishekcur · 2026-07-27
• Dynamo avoids KV routing to the “GPU that remembers” and balances load instead — Abhishekcur · 2026-07-28
• GPU scheduler estimates queued read work by unfinished tasks, not request size — Abhishekcur · 2026-07-28
• Kimi K3 serving uses relative GPU queue discounts to boost load balancing — Abhishekcur · 2026-07-28
• [source] Decoding NVIDIA Dynamo's router: Unifying GPU load and cache into one metric — Abhishekcur · 2026-07-28
• Dynamo Architecture: Pricing Memory to Optimize GPU Routing — Abhishekcur · 2026-07-28
• [source] A look at LLM inference routing: why Dynamo does not send requests to the GPU that remembers your prompt — _jaydeepkarale · 2026-07-28

1 near-duplicate retellings: Abhishekcur