FlashMask-3: Mask-Aware Distributed Attention Boosts H100 Throughput Up to 141%
pmttyji · reddit · 2026-10-20
FlashMask-3 rewrites mask-aware attention for Hopper GPUs and multi-node context parallelism.
Key improvements
- Mask-aware Hopper kernel with three-role warp specialization on FlashAttention-3's TMA/WGMMA pipeline.
- Zero-communication mask localization via a clip–shift–max on boundary vectors under any multi-chunk context-parallel assignment.
- Sparsity-aware load balancer (HCC-IPO), compute-communication overlap, and mask-driven communication pruning.
Results
- Single H100, 4K–128K context across 12 masks: TFLOPs 45–141%/34–71%/2–44% higher than FlashMask/FlexAttention/MagiAttention.
- CP degrees 4–32: throughput 1.80–3.71× over Megatron-Core CP ring.
- 118B-A14B MoE at 128K on 128 H100s: 7.3–11.0% end-to-end throughput gain, attention fwd+bwd 24.5–44.9% faster.
Open-sourced under PaddlePaddle.
More from Infra
- Victor Taelin publicly begs to buy an 8-GPU Rubin machine — 'nobody is selling it' — fforres · 2026-10-21
- Nscale joins PyTorch Foundation as Platinum member after Anyscale acquisition move — PyTorch · 2026-10-21
- Decision Models launches on Antseed with an open-source auto model router — airesearch12 · 2026-10-21
- Gradium open-sources Phonon: 31M/40M edge TTS models with Rust inference stack for 10+ backends — l-m-z · 2026-10-21
- PostgreSQL 19 slips to November as LLM-assisted code review floods committers with edge cases — sanjaykalra · 2026-10-21
- Eon cuts engineer onboarding from 2 days to 2 minutes with an MCP Gateway on Bedrock AgentCore — AWS ML Blog · 2026-10-20