MiTA: Viewing Attention as an Ever-Widening FFN Cuts Long-Sequence Training Time by 77%

新智元 · wechat · 2026-10-12

A BUPT team proposes Mixture-of-Top-k Attention (MiTA), accepted to NeurIPS 2026 and open-sourced. The key insight: standard attention can be written as an FFN whose hidden width grows with sequence length, with weights generated from the current input (fast weights). This unifies efficient attention into compression (linear attention, Linformer, TTT) and routing (block MoE, per-query Top-k), extended into a five-dimensional taxonomy.

MiTA first extracts a small number of landmark queries, which select Top-k key-value pairs from the full context to form m reusable, deformable routed experts; the same landmarks also build one global shared expert via cross-attention. Queries only pick an expert, and both branches share computation with online Softmax merging.

Results:

ImageNet-1K (DeiT-S): 79.8% Top-1, only 0.1pp below full attention; 81.2% with DWC and gating.
ADE20K: each query touches only 98 KV pairs vs 1,024–1,600 for full attention; FLOPs drop up to 42%.
Long Range Arena: 58.91% average accuracy (beating full attention), training time down from 10.6h to 2.4h (77% faster); 159x inference throughput gain at 1,024K length.
arXiv long-doc classification leads at 16K/32K/64K.

Two notable findings: emergent token pruning without any pruning module, and algorithmic generalization — after training, swapping the attention mechanism or tuning m and k at inference adjusts the compute budget with fixed weights.

Original post →

MiTA: Viewing Attention as an Ever-Widening FFN Cuts Long-Sequence Training Time by 77%

More from Research