NVIDIA researcher counters linear attention debate: 8B hybrid beat Transformer 7 months before MiniMax-01
ctnzr · x · 2026-10-18
Amid the "is linear attention a Chinese psyop" debate, NVIDIA's @ahatamiz1 responds that while scaled linear attention does take talent, the criticism doesn't hold: NVIDIA's June 2024 study (arXiv:2406.07887) trained 8B Mamba, Mamba-2 and Transformer models on identical data up to 3.5T tokens. A hybrid (43% Mamba-2, 7% attention, 50% MLP) beat the 8B Transformer on all 12 standard tasks (+2.65 avg), generated up to 8× faster, and matched or beat it on 23 long-context tasks — predating MiniMax-01 by 7 months, Qwen3-Next by 15, and Kimi Linear by 16.
More from Models
- Fixing one typo completely changes Google AI Overview's answer — krishnabrodhi · 2026-10-18
- Anthropic's Cyber Verification Program Leaked: Relaxed Safeguards for Vetted Security Researchers — arthurcolle · 2026-10-18
- Dev Dismisses Benchmarks: "I Just Want Models Actually Useful to My Use Cases" — gnukeith · 2026-10-18
- Google's AI search results suddenly got noticeably better, model routing suspected — haider1 · 2026-10-18
- Context windows grew 250x but cross-session memory is still unsolved — khademinori · 2026-10-18
- Astra AI Refuses Request Because It "Might Enable Your Research Goal" — cephaloform · 2026-10-18