AWS details fused MoE kernel engineering on Hopper, yielding 2-4% throughput gains
AWS ML Blog · rss · 2026-10-13
An AWS ML Blog post breaks down kernel optimizations for MoE models on SageMaker. First, a Swap-AB matrix transpose works around Hopper's WGMMA M=64 operand layout constraint (decode-phase batches are often M=1), filling Tensor Cores and enabling async prefetching—delivering 2-4% end-to-end throughput gains on H200, upstreamed to vLLM 0.24.0+. Second, profiling shows the Triton backend consistently beats the default FLASHINFERCUTLASS path for FP8 tensor-quantized MoE at low concurrency, thanks to single-pass register-level fusion that reads HBM once.
More from Infra
- Microsoft, AWS, GCP may hit $100-120B AI revenue in 2026, doubling YoY — BenBajarin · 2026-10-14
- Prime Intellect, Valued at $1B, Acquires Moondream to Speed Up Open Model Inference — himanshustwts · 2026-10-14
- Applied Materials CEO: "We Manufacture Miracles All the Time" — BenBajarin · 2026-10-14
- Cloudflare D1 Scales to 5.5 Million Databases: Inside the Control Plane Architecture — Cloudflare Blog · 2026-10-14
- Gavin Baker: open-weight tokens will dominate, B200 clusters still yield 37% IRR — GavinSBaker · 2026-10-14
- AMD's Lisa Su sees strong AI inference and CPU demand at OCP summit — firstadopter · 2026-10-14