SGLang's Breakable CUDA Graph speeds prefill graph building by 3.8–5.2x
ying11231 · x · 2026-09-05
The SGLang team published a fully visualized deep dive on Breakable CUDA Graph (BCG), open-sourced in February, which keeps CUDA Graph working around ops it cannot capture.
- BCG captures graphs where traditional CUDA Graph can't, by splitting around non-capturable ops
- Became SGLang's default prefill path in April; the diffusion stack adopted it in July
- Numbers: prefill graphs build 3.8–5.2x faster in a quarter of the code; prefill runs 1.70x over eager with BCG, 1.93x with full capture
- Built with AI at Meta, with help from NVIDIA, AMD, PyTorch, and Thinking Machines
More from Infra
- Running an Opus-level coding agent locally at 2x speed for free: a 15-page report — julianharris · 2026-09-05
- There's no agreed way to value a GPU running inference—and compute futures now settle on these indexes — AccBalanced · 2026-09-05
- AI Now on data center boom: community pushback and 'they won't build them where they live' — AINowInstitute · 2026-09-05
- Gemma 4 Runs 151.4% Faster on Mac via Community MLX Inference Optimization — gajesh · 2026-09-05
- Qwen3.8-Flash-Next runs fully locally on a Xiaomi 14T Pro phone CPU — Tall_Abrocoma_3533 · 2026-09-05
- SemiAnalysis: OpenAI's ASIC program is leverage — Altman wins even if the chip loses — MarvinTBaumann · 2026-09-05