Muir linear attention streams 10M tokens in 25MB state, trains 2.7x faster than Transformer at 32K
_akhaliq · x · 2026-11-07
Researchers from Google released Muir (arXiv 2611.08376), a linear attention architecture with three co-designed pieces:
- Sensitivity-allocated low-rank gating: state updates are far more sensitive to the erase gate than the write gate, so erase is parameterized at low rank; freed parameters go into a 30.4% wider FFN at iso-params.
- Matched-filter readout: suppresses interference from slow-decaying state rows that accumulate 100x larger norms, boosting 5-hop variable tracking root retention by 19.6 points at 1.34B.
- Division-free block-Neumann kernel: stable parallel update suited to wide-matrix-unit accelerators like TPUs.
Results: perplexity drops 1.8–2.0% vs GDN-2 and 2.5–6.0% vs Transformer across 166M–1.34B; constant 25MB state streams 10M tokens without drift with near-100% 1M passkey recall; training is 2.7x faster than Transformer at 32K.
Related event: Google Unveils Muir Linear Attention Architecture(3 posts)→
More from Research
- ICLR 2027 Score Atlas: check your review score percentile across 40k+ submissions — _vztu · 2026-11-07
- 3D reconstruction shows what the world looks like; world models show how it behaves — prateekj · 2026-11-07
- A 4B model trained on one GPU in 2 hours beats Jev 1.13 at agent decisions, 86.5% vs 80.6% — ivan_bezdomny · 2026-11-07
- XM4D: certifiable geometric optimization for 4D reconstruction from monocular video — kwangmoo_yi · 2026-11-07
- StepFun's StepX-OPD merges capability learning and few-step distillation for diffusion models — stepfun-ai · 2026-11-07
- Goodfire study: probes break predictably under retraining—concepts move, not vanish — mathildepapillo · 2026-11-07