GPU routing uses queue-aware memory discounts instead of a fixed reuse rule
Abhishekcur · x · 2026-07-27
A routing trick for GPU-backed inference systems gives memory reuse a dynamic discount instead of treating it as an absolute rule.
- The discount shrinks as a GPU’s queue gets longer.
- But the queue is measured against the least busy GPU you could send the request to, not against zero.
- That means the currently freest GPU keeps its full memory advantage.
- Queue length is also compared to the size of the request, so large requests can tolerate longer waits than small ones.
- The setting is disabled by default and is meant for cases where reuse matters, but not at any cost.
Related event: How NVIDIA Dynamo Prices LLM GPU Routing Instead of Hard KV Rules(14 posts)→
More from Infra
- Dedicated Hardware for Running AI Agents at Scale Arrives — cyrilzakka · 2026-09-23
- Ternary Bonsai 2 27B: 5.9GB weights retain ~95% of full-precision reasoning — cephaloform · 2026-09-23
- Qwen 27B runs 24hr unattended on one RTX5090, builds full Postgres-SpringBoot-React spreadsheet app — anglepoiselife · 2026-09-23
- OpenRoboto Shift launches: decentralized egocentric video data network for robot brains — markjeffrey · 2026-09-23
- Engineer describes designing digital circuits that recycle most of their energy — MikePFrank · 2026-09-23
- Cloudflare CTO Dane Knecht makes TIME's 2026 executives list as AI crawlers hit 52% of traffic — dinasaur_404 · 2026-09-23