Megakernel Loses to CUDA Graph + Multi-stream + PDL in Real GLM-5.2 Decode Layer Optimization on B300
青稞AI · wechat · 2026-10-09
A hands-on engineering deep-dive optimizing the GLM-5.2-FP8 decode Indexer layer on a B300 GPU: 30 short kernels taking 80us dropped to 61.6us via fusion, a custom Megakernel (MegaRTP) reached 39.6us, but a CUDA Graph + Multi-stream + PDL stack won at 36.992us.
- Post-fusion single-stream SM utilization cap was only 43.7%; the bottleneck is serialized scheduling, not launch bubbles (1% of runtime)
- Multi-stream parallelizes independent branches (61.6→44.4us); PDL lets consumers do prologue and weight prefetch early (→36.99us), but gains depend on how much useful work can run ahead — plain early entry saves under 1us
- Megakernel loses due to: no safe 2-CTA GEMM pairing, global-counter polling hitting 4000-cycle latency under HBM pressure, and bandwidth contention from aggressive prefetching
- Author's hot take: Megakernel's real edge is tile-level dependency and cross-instruction pipelining, only realized in workloads like MegaMoE with abundant cross-wave fine-grained dependencies; its prevalence at batch=1 reflects favorable fixed-gain ratios, not semantics
Includes detailed analysis of Stanford Megakernels, Mirage, and TileRT — a high-quality reference for inference kernel engineering.
More from Infra
- Cadence announces sovereign AI cloud for European chip designers — karlfreund · 2026-10-23
- SGLang Deep-Dive Blog Series Breaks Down KV Cache, Posts up to 15.5x Throughput Gains — BanghuaZ · 2026-10-23
- DynaLM: Open-Source CPU-First LLM Inference Engine Seeks Feedback — Unlikely_Safety_7456 · 2026-10-23
- Capital Is Now the AI Buildout Bottleneck, Says Analyst Citing Jensen Huang — JOBhakdi · 2026-10-23
- Cerebras explains how it deploys trillion-parameter models on wafer-scale chips — bookwormengr · 2026-10-23
- IFM open-sources RL360, the agentic RL infra behind K2 Horizon's training — rupspace · 2026-10-23