Google's Muir linear attention fits 10M tokens in a 25MB state, beats GDN-2
CategoryNormal149 · reddit · 2026-11-07
Google Research published Muir, a linear-attention architecture that streams 10M tokens in a fixed 25 MB state — versus a 2.1 TB KV cache for a 1.3B Transformer at the same length (0.32 TB even at 2.5-bit quantization).
The paper tackles three bottlenecks that channel-wise gated delta models like GDN-2 hit over 100B-token training:
- 100x slow-channel glare: a matched-filter readout (inverse fractional power law + per-token query modulation) relaxes write erasure by 17% and extends lookback by 60%.
- 13x write-vs-erase sensitivity: fusing all 5 gates into one rank-512 projection cuts gate params 4.5x, widening the FFN by 30.4% at a strict parameter budget.
- Numerically stable large-chunk kernel: boundary-pivoted multiplicative decays avoid fp32 underflow (previously NaNing 25% of heads), and a block-Neumann solver routes 98% of chunkwise FLOPs to matrix units.
Results: beats GDN-2 and Transformers on perplexity and zero-shot reasoning at every scale (1.2x token efficiency vs. GDN-2), trains 2.7x faster than Splash Attention at 32K; lifts RULER 5-hop Variable Tracking retention from 28.4% to 48.0% and 8K early-depth recall from 1% to 31%; hits 90.0% zero-shot (99.4% post-cooldown) passkey accuracy at 1M tokens, where GDN-2 collapses past 4K-8K and Transformers OOM past 64K.
Related event: Google Unveils Muir Linear Attention Architecture(3 posts)→
More from Research
- Epoch AI Researchers Push for Evidence Transparency in AI R&D Speedup Report Estimating ~2x Acceleration — Jsevillamol · 2026-11-07
- Humanoids at the Edge Workshop Extends CFP to Nov 12, Focuses on Humanoids in Extreme Environments — omarsar0 · 2026-11-07
- Turn gaps average ~200ms: duplex voice models must plan replies mid-listen — smolix · 2026-11-07
- Why Higgs Realtime Duplex uses a 160ms step: it matches human decision speed — smolix · 2026-11-07
- ByteDance experiments suggest Full Bandwidth Transformer matches models twice its size — DimitrisPapail · 2026-11-07
- Ai2 brings 10+ robotics papers to CoRL 2026, spotlighting MolmoAct2 and MolmoBOT — allen_ai · 2026-11-07