GPU scheduler estimates queued read work by unfinished tasks, not request size
Abhishekcur · x · 2026-07-28
The thread explains one subtle part of a GPU scheduler: when estimating read work, it does not count how much the request itself needs to read.
Instead, it counts how much reading the GPU still owes: all already-accepted requests that are unfinished, plus the new request. In other words, the system asks not “how big are you?” but “how long is the line you’re about to join?”
That single change makes the scheduler behave more like a real load balancer instead of a simple lookup.
Related event: How NVIDIA Dynamo Prices LLM GPU Routing Instead of Hard KV Rules(14 posts)→
More from Infra
- Together AI adds canary rollouts for zero-downtime model upgrades on dedicated inference — togethercompute · 2026-09-23
- Dedicated Hardware for Running AI Agents at Scale Arrives — cyrilzakka · 2026-09-23
- Ternary Bonsai 2 27B: 5.9GB weights retain ~95% of full-precision reasoning — cephaloform · 2026-09-23
- Qwen 27B runs 24hr unattended on one RTX5090, builds full Postgres-SpringBoot-React spreadsheet app — anglepoiselife · 2026-09-23
- OpenRoboto Shift launches: decentralized egocentric video data network for robot brains — markjeffrey · 2026-09-23
- Engineer describes designing digital circuits that recycle most of their energy — MikePFrank · 2026-09-23