GPU scheduler estimates queued read work by unfinished tasks, not request size
Abhishekcur · x · 2026-07-28
The thread explains one subtle part of a GPU scheduler: when estimating read work, it does not count how much the request itself needs to read.
Instead, it counts how much reading the GPU still owes: all already-accepted requests that are unfinished, plus the new request. In other words, the system asks not “how big are you?” but “how long is the line you’re about to join?”
That single change makes the scheduler behave more like a real load balancer instead of a simple lookup.
Related event: Dynamo GPU Scheduler Scores by Outstanding Read Workload(2 posts)→
More from Infra
- 800V DC data centers still need standby power, MS note says — BenBajarin · 2026-07-28
- Gary Marcus says investors are seeing through Nvidia’s data-center backstop play — GaryMarcus · 2026-07-28
- Kimi K3 lands on Fireworks AI for inference and training — omarsar0 · 2026-07-28
- YC expands AI startup support with $25k compute credits and a GPU cluster — ycombinator · 2026-07-28
- vLLM Collaborates with DigitalOcean to Host Kimi K3 Model — vllm_project · 2026-07-28
- Amazon and Microsoft will each spend about $200B on AI data centers this year — fortune · 2026-07-28