GPU veteran: mega kernels only win at batch-1 inference, CUDA graphs still reign
mike64_t · x · 2026-11-11
A technical take on GPU kernel engineering: for specific workloads like batch size 1 inference, mega kernels (fusing everything into one kernel) are likely optimal since any graph involvement hurts; but this does not generalize — it stops being true at larger scales and especially training.
The author pushes back on the community extrapolating "mega kernels win here" into "a mega kernel would be faster for any workload." The conservative default: hardware kernel-scheduling acceleration exists, and not using it — or using it at the wrong granularity — is what costs you. In practice, CUDA graphs with co-designed kernels remain the superior programming paradigm.
Related event: Megakernel Route Sparks Heated Debate in CUDA Community(6 posts)→
More from Infra
- EE jokes about AI progress chart: training compute growing 10.6 dB per year — jwt0625 · 2026-11-11
- Used A100 SXM4 at €2,700 vs New Radeon 9700: A Home Lab Buyer's Dilemma — Hildechis · 2026-11-11
- Cloudflare announces OpenAI Codex can now run on Durable Objects like it runs on your laptop — tekbog · 2026-11-11
- Musk: SpaceX in-house turbine blade casting could cut AI data center power bottleneck by 18 months — elonmusk · 2026-11-11
- Musk says SoftBank is defaulting on turbine orders after stockpiling power of six Hoover Dams — elonmusk · 2026-11-11
- Cheaper Inference pools demand to slash AI inference costs 40.5% across 502 models — NandoDF · 2026-11-11