Dynamo routes by total cache-block cost, not by prompt memory alone
Abhishekcur · x · 2026-07-27
Cache-aware routing is about more than just memory
The thread explains why Dynamo does not simply route a request to the GPU that already holds the prompt cache.
- A GPU with the cached prompt may already be overloaded.
- Dynamo turns queueing work, occupied space, and cache reuse into the same unit: cache blocks.
- The router then computes one score per GPU and picks the smallest one.
- Cache locality is treated as a discount, not a hard routing rule.
The key idea: remembering a prompt reduces cost, but it does not automatically win the routing decision.
Related event: How NVIDIA Dynamo Prices LLM GPU Routing Instead of Hard KV Rules(14 posts)→
More from Infra
- Dedicated Hardware for Running AI Agents at Scale Arrives — cyrilzakka · 2026-09-23
- Ternary Bonsai 2 27B: 5.9GB weights retain ~95% of full-precision reasoning — cephaloform · 2026-09-23
- Qwen 27B runs 24hr unattended on one RTX5090, builds full Postgres-SpringBoot-React spreadsheet app — anglepoiselife · 2026-09-23
- OpenRoboto Shift launches: decentralized egocentric video data network for robot brains — markjeffrey · 2026-09-23
- Engineer describes designing digital circuits that recycle most of their energy — MikePFrank · 2026-09-23
- Cloudflare CTO Dane Knecht makes TIME's 2026 executives list as AI crawlers hit 52% of traffic — dinasaur_404 · 2026-09-23