FlashAttention vs standard attention in one chart: the bottleneck is data movement, not compute
cneuralnetwork · x · 2026-10-29
A comparison chart explains why FlashAttention is faster:
- Standard attention materializes large N×N intermediate matrices (S=QKᵀ, P=softmax(S), O=PV) in GPU memory, causing heavy read/write traffic.
- FlashAttention loads small Q/K/V tiles (e.g. 64×64) into fast on-chip SRAM, computes attention per tile with numerically stable online softmax, and accumulates partial outputs on-chip—no large intermediate matrices.
- Key insight: the bottleneck isn't computation, it's moving data.
- Result: far less memory traffic, exact same attention output, faster and more memory-efficient.
More from Infra
- Prime Inference scales from zero to 120B tokens per day on OpenRouter in one month — xeophon · 2026-10-29
- SJTU's TAO Cuts LLM Agent Serving Latency Up to 26.8% With Tool-Aware Joint Scheduling — AIOR-Research · 2026-10-29
- Nscale's unproven CEO chases $30 billion neocloud IPO amid AI turmoil — bloomberg · 2026-10-29
- Qwen3.8 Flash Next at IQ3_S hits 275 tok/s on a single RTX 5080 via Basalt fork — yehyakar · 2026-10-29
- Google Cloud CEO: Apple Scaling Foundation Model Pretraining on Latest TPUs — shayan_ · 2026-10-29
- Particle-1 inference goes live on OpenRouter, claiming 28-52% fewer output tokens — ycombinator · 2026-10-29