Dynamo turns GPU routing into one cache-block score instead of separate heuristics

Abhishekcur · x · 2026-07-28

This thread explains a GPU routing system that turns queueing, memory reuse, and runtime space into one common unit so it can pick the best GPU with a single score.

Core idea

A GPU is not chosen by separate rules like “check memory first, then load.”
Instead, everything is converted into cache blocks and summed into one number per GPU.
The router then simply picks the GPU with the smallest total cost.

How the math works

Work in line is defined as the amount of work the GPU still owes, not the size of the incoming request.
Space is the capacity already consumed by running requests plus what the new request would add.
Memory reuse becomes a discount, but the discount depends on where the cached state lives:
on GPU memory: full discount
in host RAM: smaller discount
on disk: smaller still

Important nuance

Reusing a prompt can make a GPU cheaper to route to, but it does not mean the router ignores load.
Cache hits are valuable because they save both reading time and new memory use, but only if the GPU still has capacity.

The overall message is that Dynamo treats routing as a unified optimization problem rather than a stack of ad hoc heuristics.

Original post →

Dynamo turns GPU routing into one cache-block score instead of separate heuristics

More from Infra