13 attention mechanisms AI engineers must know, organized by the bottleneck each solves
blaizedsouza · x · 2026-09-13
@akshaypachaar compiled 13 attention mechanisms AI engineers must know, organized by the actual bottleneck each solves rather than discussing them as one blob.
The framework:
- KV cache size bottleneck → MHA (max flexibility, largest cache), MQA (all query heads share one KV head), GQA (grouped sharing, a balance)
- Others cover which tokens can attend to each other, cheaper attention computation (FlashAttention, Sparse attention), and KV cache management during serving (PagedAttention, RadixAttention)
The original article illustrates Self-attention, Cross-attention, Multi-Head Attention and more — worth bookmarking as a reference sheet.
More from Infra
- PicoLM runs a 1B-parameter LLM on a $10 board with 256MB RAM — pure C, zero dependencies — tom_doerr · 2026-09-14
- AirLLM streams model layers one at a time: 70B LLM on a 4GB GPU, 2.8T Kimi K3 under 4GB VRAM — alex_verem · 2026-09-14
- JAX-QNN v0.1.0: Open-Source PJRT Backend Runs JAX Natively on Snapdragon — carrycooldude · 2026-09-14
- Memory now 63% of AI accelerator cost, up from 52% in early 2024 — Summit-Star001 · 2026-09-14
- Ollama's jmorgan: small models now handle most conversational and reasoning use cases — ollama · 2026-09-14
- Chinese Optical Transceiver Makers Dodge US Blacklist; Zhongji Innolight Jumps 4% — pstAsiatech · 2026-09-14