Nested CUDA Graphs in PyTorch 2.14 deliver 4.5x steady-state speedup on B200 at 33% cold-start cost

@cfregly published a set of compiler microbenchmarks and a mechanism analysis of the nestedgraphbreaks feature in PyTorch 2.14.1 (CUDA 13.0) on the NVIDIA B200: with the same 64 operators (1,048,576 FP32 elements), enabling nested graphs cut the synchronized steady-state median latency in the depth-8 scenario from 407.57µs to 89.94µs, with improvements across all paired combinations and speedups of 4.50×–4.58×.

Confirmed

Mechanism: comparing the depth-8, seed-0 Inductor-generated code, without nested graphs there are 16 wrapper regions × 4 ATen operators, versus 2 regions × 32 operators with them enabled—both covering the same 64 operators
The number of CUDA Graph replay launches per profiled call drops from 16 to 2; nested graphs also produce less memory fragmentation and larger fusion regions, which explains the sharp drop in steady-state latency
Cold-start cost: with the feature enabled, the total time of the first 10 synchronized compiled calls rises from 2.411s to 3.212s (+33.2%), with the extra time almost entirely concentrated in the first call (roughly 0.79–0.88s more), falling off noticeably from the second call onward

Why It Matters

This is a classic "big steady-state win, small cold-start loss" optimization: a clear gain for long-running, repeatedly executed hot paths, but a trade-off for short-lived or frequently cold-starting workloads
The author stresses measurement methodology: inspect the compiler-generated code snippets and CUDA replay activity, and measure first-call cold-start latency separately from steady-state latency of repeated executions—otherwise the true effect of such optimizations is easy to misread

2026-10-22 ~ 2026-10-22 · 6 related posts

Primary sources

Nested graphs deliver 4.5× steady-state speedup in depth-8 benchmark — cfregly · 2026-10-22 12:29 (UTC+08:00)
Nested CUDA graphs trade 33.2% longer cold start for faster steady state — cfregly · 2026-10-22 12:30 (UTC+08:00)
PyTorch nested_graph_breaks gives 4.53× steady-state speedup on B200 — cfregly · 2026-10-22 12:29 (UTC+08:00)

[source] PyTorch nested_graph_breaks gives 4.53× steady-state speedup on B200 — cfregly · 2026-10-22
[source] Nested graphs deliver 4.5× steady-state speedup in depth-8 benchmark — cfregly · 2026-10-22
Nested graphs merge 16 Inductor regions into 2 for the same 64 ops — cfregly · 2026-10-22
Nested CUDA graphs cut replay launches from 16 to 2 per call — cfregly · 2026-10-22
[source] Nested CUDA graphs trade 33.2% longer cold start for faster steady state — cfregly · 2026-10-22
How to benchmark compiler opts: inspect emitted code, split cold vs steady — cfregly · 2026-10-22