PyTorch nested_graph_breaks gives 4.53× steady-state speedup on B200
cfregly · x · 2026-10-22
On an NVIDIA B200 with PyTorch 2.14.1 (CUDA 13.0), setting nestedgraphbreaks=True cut steady median latency for the same 64 ops (1,048,576 FP32 elements) from 407.57µs to 89.94µs — a 4.53× speedup (4.50×–4.58× across paired seeds).
The mechanism: Inductor regions merge from 16×4 ops into 2×32 ops, cutting CUDA Graph replay launches from 16 to 2 per call, with less fragmentation and larger fused regions. The tradeoff is cold start: the first ten calls took 33.2% longer overall (2.411s→3.212s), almost all in call 1. Author's advice: try nestedgraphbreaks=True now, inspect emitted compiler code and replay activity, and measure cold first use separately from steady state.
More from Infra
- HUMAIN and NVIDIA plan up to 84MW of AI compute in Saudi Arabia, ~27,000 GPUs — HUMAIN · 2026-10-22
- Jensen Huang: Energy is AI's biggest bottleneck, time to invest in all forms of energy production — kimmonismus · 2026-10-22
- NVIDIA unveils Vera Rubin full-stack agent hardware, claims post-trained Nemotron beats frontier models — mervenoyann · 2026-10-22
- Meme: all of modern digital infrastructure rests on one ASML machine — VoidAsuka · 2026-10-22
- OrbStack adds self-hosting for orbs, free for individuals and teams — jasonkneen · 2026-10-22
- Forward-only pretraining matches backprop: val 3.704 in 2.5 min on 8×H100, no autograd — tensorqt · 2026-10-22