Understanding FlashAttention: A Handbook Tracing FA1 to FA4 and Why HBM Traffic, Not FLOPs, Is the Bottleneck
techNmak · x · 2026-09-10
A new handbook in the "Understanding AI" series explains FlashAttention as a systems problem: the math is highly parallel, but naive implementations materialize N×N intermediates in HBM. FA tiles Q/K/V into on-chip memory with online softmax, keeping the same O(N²d) dense computation — gains come purely from cutting memory movement, even recomputing scores in backward when that's cheaper than HBM traffic. It covers FA2's work partitioning, FA3's Hopper async + FP8, FA4 on Blackwell, PyTorch SDPA, common implementation mistakes, and why PagedAttention (KV-cache management) is a different thing.
Related event: Understanding FlashAttention: Why Memory Traffic Beats FLOPs(2 posts)→
More from Infra
- PlanetScale launches Neki, a sharded Postgres for workloads outgrowing single machines — threepointone · 2026-09-10
- Five key levers for controlling AI costs, and why that advice may soon expire — rseroter · 2026-09-10
- Positron raises $875M at a $5B valuation, per WSJ — JacquesThibs · 2026-09-10
- Meetas: fully local meeting assistant grounds every claim in transcript evidence, runs a 27B Q4 model offline — Odd-Name-1556 · 2026-09-10
- NVIDIA's BioNeMo Inference Runtime hits public beta, boosting Boltz-2 folding throughput 2.9x — AllThingsApx · 2026-09-10
- US PCB production fell from ~40% to ~4%: an interactive atlas traces one board's supply chain — AnneliesGamble · 2026-09-10