The Softmax Wall: in FP8/FP4 attention is no longer bottlenecked by matmuls
burny_tech · x · 2026-11-10
Vincent Zhong's deep-dive argues chipmakers doubled matrix throughput per generation without scaling SFUs, vector ALUs, or shared-memory bandwidth, inverting the attention roofline: at FP8/FP4, softmax's transcendental math and on-chip score traffic dominate.
Key points:
- On AMD MI355X, the two FP8 attention GEMMs for a 32×128 tile take 1024 cycles, but exact FP8 softmax alone takes 896 cycles;
- Blackwell emulates transcendentals via polynomial FMA; Rubin needs a 7-step TMEM repack round-trip;
- 2:4 structured sparsity collapses pairwise at 4:8, so fine-grained sparsity is a trap;
- Coarse token-level sparsity — dropping whole keys before softmax, as in DeepSeek Sparse Attention and DeepSeek-V4.1 — is the viable path.
More from Infra
- Hetzner's eBPF-based hcnet network stack hits beta with 240k concurrent connections per server — DanielLockyer · 2026-11-10
- New Export Control Push After Spy Scandal Adds to Nvidia's Headaches — pstAsiatech · 2026-11-10
- You can download a model but not a substation: investor case for AI compute bottlenecks — McDonaghMatthew · 2026-11-10
- Lokey: Open-Source Android Keyboard Runs On-Device AI Prediction and Translation — Nicolodeva · 2026-11-10
- Report: Moonshot AI obtained Nvidia's advanced chips this summer; skeptic says not a serious training cluster — pstAsiatech · 2026-11-10
- Data center investment up 21% in a year, one of only two US investment booms — soumitrashukla9 · 2026-11-10