B200 kernel shootout: Megatron + CUDA graphs at 43.1s beats custom tcgen05 kernel's 48.1s
akbirthko · x · 2026-11-03
realarnoldxu shares a kernel benchmark training a 162M-param GPT for 4,000 steps on 116 SMs of a B200 (each baseline run alone via CUDA MPS, same data/init weights/batch order, final loss 2.1575):
- Megatron + TE + CUDA graphs: 43.1s (fastest)
- open-flapping-airplanes custom tcgen05 kernel: 48.1s (their kernel: 46.1s)
- PyTorch + flash-attn graph: 50.9s; eager: 52.5s
- Megatron + TE eager: 62.4s (their pipeline 34% slower, theirs 23%)
Takeaway: CUDA graphs deliver more speedup than the custom kernel's edge — even the winning hand-written kernel loses to the CUDA-graphs Megatron baseline.
More from Infra
- Anthropic's signed leases hit ~$321B as purchase commitments jump from $55B to $266B — tengyanAI · 2026-11-04
- Stanford's ParallelKernelBench lands NeurIPS 2026: LLMs nail only 32% of multi-GPU kernels — simonguozirui · 2026-11-04
- SpecPilot: Berkeley and NVIDIA's Agent Loop Auto-Optimizes Speculative Decoding per Deployment — UCBerkeley · 2026-11-04
- AMD exec joins Fireworks AI Forge event to highlight multigenerational partnership — AnushElangovan · 2026-11-04
- Microsoft details Outlook search overhaul: 7 relevance-ranked results and offline resilience — benjamingauthey · 2026-11-04
- AMD exec gifts DHH a 192GB AMD workstation, embraces Omarchy Linux — AnushElangovan · 2026-11-04