NVIDIA veteran explains why CUDA Graphs have lower launch overhead than streams

On October 21, joefioti asked NVIDIA why enqueueing kernel launches into a stream costs more than using a CUDA graph—if the CPU can run ahead of the GPU, the overhead theoretically shouldn't exist. The question drew several deep answers on CUDA's dispatch internals in the same batch.

Confirmed

• blelbach, who worked on early CUDA Graphs development, gave the low-level explanation: everything comes down to writing a command buffer and sending it to the GPU. With streams, every launch has to rebuild that buffer—allocating memory, encoding information, and sending it. With graphs, the graph is compiled up front, completing allocation and encoding ahead of time and eliminating the repeated per-launch cost. joefioti relayed this "precompiled buffer and encoding" explanation as well.
• mike64t added the QMD (queue metadata descriptor) angle: on the stream path, cuLaunchKernel must dispatch QMDs on the spot and maintain a tracking system for in-flight QMDs—a design cost of "dispatch on demand."
• mike64t also noted that QMDs produced on the batched launch path carry less real dependency information, while the stream path's QMDs carry more, further adding to construction and tracking costs.

Why it matters

• This discussion takes "why graphs are faster" from an empirical rule down to the concrete mechanics of command buffer construction, QMD encoding, and dependency tracking—directly useful for understanding where CUDA launch overhead comes from and for choosing between graphs and streams in high-frequency, small-kernel scenarios.

2026-10-21 ~ 2026-10-22 · 5 related posts

• [source] Developer Asks NVIDIA Why Stream Kernel Launches Cost More Than CUDA Graphs — joefioti · 2026-10-21
• [source] CUDA vet explains why CUDA Graphs beat streams on kernel launch overhead — blelbach · 2026-10-21
• Why CUDA Graphs Beat Streams on Launch Overhead: Precompiled Buffers and Encoding — joefioti · 2026-10-21
• [source] Why CUDA Stream Kernel Launches Cost More Than Graphs: QMD Construction and Tracking — mike64_t · 2026-10-22
• The cost of instant dispatch in CUDA: stream-path QMDs carry less dependency information — mike64_t · 2026-10-22