The Evolution of Attention in Large Language Models: Mechanisms, Trade-offs, and Emerging Trends
Zhentao Tan, Jingyi Shen, Yanbo Li, Yao Liu, Yue Wu, Jieping Ye
cs.CL
2026-09-30
Alibaba's survey reads softmax, sparse, linear, SSM, and hybrid attention as contextual memory via five dimensions; 59 model releases show divergence, not convergence.
Research on attention efficiency has split into lines that barely talk to each other: KV-cache compression, sparse access, methods that fold history into recurrent states (linear attention, SSMs), and hybrids that compose several mechanisms in one model. Each line has its own vocabulary, and existing surveys organize by architecture family, computational complexity, or long-context application, so explicit-memory compression and recurrent-state construction have never shared one comparison language. This survey from Alibaba's Token Hub team, covering public work through September 22, 2026, does two things: propose a unified lens, and reconstruct what major models actually ship.
One lens plus one inventory.
The lens treats every sequence mixer as a system of model-internal contextual memory, meaning input-dependent information available while processing a sequence, excluding pretrained parameters and external retrieval. It splits memory handling into five functional dimensions:
These are functional labels, not five implementation blocks; one operation can serve several roles. Applied across research lines, the division of labor is sharp: Softmax Attention innovates on Representation and Update (GQA shares KV across query heads; MLA compresses KV into a low-rank latent), Sparse Attention on Access, where candidate-set construction is the central problem, and linear attention and SSMs on constructing and updating recurrent states.
The inventory covers 59 release-level records across 14 major model lineages, plus a frozen comparison of 11 high-performing open-weight endpoints, all closed on September 22, 2026. Architectures with insufficient disclosure are marked undisclosed rather than inferred.
The survey reports no benchmark scores; its findings sit at three levels:
| Level | Finding |
| Mechanism | Explicit-memory and recurrent-state methods keep different interfaces while controlling an increasingly overlapping set of memory functions |
| Architecture | Designs remain heterogeneous; coordination increasingly runs along network depth, via layer-wise composition and cross-layer reuse of memory and routing artifacts |
| Outlook | A stateful multidimensional memory-routing hypothesis: persistent memory organized across temporal scope, depth, substrate, and granularity, governed by coordinated Sparse Write and Sparse Read |
Concrete inventory facts: GQA covers Llama 3/4 and Qwen2/3, and survives as a periodic gated full-attention path in Qwen3.5, Qwen3.6, and Qwen3.8-2.4T-A95B. MLA, introduced in DeepSeek-V2 and kept through V3/V3.2, is used by Kimi K2, appears as Gated MLA in Kimi K3, and serves as the content representation under DeepSeek Sparse Attention in the GLM-5 series. DeepSeek-V4 attends densely over heavily compressed entries. On the sparse side, MiniMax-M3 (about 428B total, 23B activated parameters) runs its block-sparse MSA in 57 of 60 layers. Across the 11 endpoints, the survey finds continued architectural heterogeneity and a persistent role for explicit token retrieval; compressed-state designs have not displaced it at the frontier.
For practitioners this is a map, not a single technique. The lens is a practical reading tool: quite a few "new attention" proposals modify only Integration (head routing, output gating), saving neither KV storage nor score computation, and the survey is explicit that numerical sparsity becomes real savings only when kernels can skip the work. The 59-record inventory is a snapshot of industrial adoption as of September 2026: the frontier mixes GQA/MLA, sparsity, and hybrids, with no single operator winning. The closing thesis, that sequence architecture design is shifting from optimizing an isolated Attention operator toward organizing the lifecycle and selective use of contextual memory, matches what NSA, DSA, and cross-layer reuse actually did in 2025-2026. It is a survey: no new mechanism, no new numbers; the contribution is the organization and the inventory.
Stated by the authors: coverage is structured but not exhaustive; the inventory is purposively curated, not market-share weighted, and explicitly cannot attribute model quality causally to an attention mechanism; modality-specific components of multimodal models are excluded. Open points on a close read: the five dimensions are admitted to be non-orthogonal, most methods touch several at once, and the discriminating power of the multi-label description is unverified; the memory-routing hypothesis is a synthesis with no testable experimental design attached; the selection rule for the 14 lineages and 59 records is described only as 'curated,' leaving room for selection bias.