Why CUDA Stream Kernel Launches Cost More Than Graphs: QMD Construction and Tracking
mike64_t · x · 2026-10-22
Explaining why enqueuing kernel launches to a stream has more overhead than a CUDA graph, the author points to the stream path's need to construct QMDs (queue metadata) on the spot and maintain a tracking system for in-flight QMDs — the price of instant dispatch, since cuLaunchKernel must dispatch the QMD right then. He finds the design odd and says a dedicated cuStreamCommence() would have been preferable, but never got an answer from NVIDIA's Nicholas.
Even with a hypothetical cuLaunchKernel plus cuCommence API, a bad implementation would still build QMDs immediately into a staging buffer or FIFO, leaving dependent kernels unknown. Tracking launches in a different representation and paying QMD construction at commence time would avoid this.
More from Infra
- July 28 MCP release makes remote servers stateless, biggest revision since launch — bibryam · 2026-10-23
- MCP keynote: AAIF hits 305 members, Skills over MCP and Tasks/Events roadmap revealed — bibryam · 2026-10-23
- Only 8% of cloud vendors support OIDC — dev calls to kill long-lived API keys — cramforce · 2026-10-23
- Swift-1.5-Qwen3.8-27B ported to hipfire MQ4: 2.2x faster end-to-end than GGUF on a 7900 XT — Queasy_Asparagus69 · 2026-10-23
- Intel CTO: chip output can at best 2X while AI compute grows 3X per year — SammaVaco · 2026-10-23
- Engineer reverse-engineers projector TV's 10s wake lag to a cold-booting Linux board — banteg · 2026-10-23