From Bahdanau to FlashAttention: a timeline of attention's 8-year evolution
cneuralnetwork · x · 2026-11-11
A tweet with an infographic traces the evolution of attention in deep learning: Bahdanau's soft attention (2014) for seq2seq alignment, Luong's global/local refinements (2015), the Transformer making self-attention the default (2017), Linear Attention tackling quadratic complexity at O(n) (2020), and FlashAttention's IO-aware exact attention that is faster and more memory-efficient (2022). The author asks which breakthrough had the biggest impact.
More from Research
- ARLoop releases fully open looped transformer recipe; 1.2B model beats Huginn-3.5B — Lianhuiq · 2026-11-11
- UniPath (NeurIPS 2026): adaptively coordinating understanding and generation in one model — jindong_wang92 · 2026-11-11
- Scale AI's 1,200-run study: PPO wins at small batches, GRPO catches up at scale — heghbalz · 2026-11-11
- Open-source effort designs personalized pancreatic cancer vaccine for 34-year-old patient — jajoosam · 2026-11-11
- Ross Girshick posts CVPR slides recounting 1.5 decades of object detection and its link to the LLM era — CatAstro_Piyush · 2026-11-11
- KAIST's WikiTrace: evidence-linked memory fixes agent "memory-induced degradation" — dair_ai · 2026-11-11