MoVA explained: how IFM's 36B-A4B matches a dense 32B with 4B active params
rohanpaul_ai · x · 2026-09-11
K2 Horizon's efficiency comes from Mixture-of-Value Attention (MoVA), which extends MoE routing from FFN layers into attention itself.
- The 36B model activates only 4B params per token yet lands close to the dense 32B
- Because the dense and sparse models were trained under identical conditions, they offer a clean dense-vs-sparse comparison instead of comparing unrelated models
More from Models
- Tencent releases Hy-SimulMT: real-time speech-to-speech translation with ~3s latency — _akhaliq · 2026-10-28
- Meta's Omnilingual MT covers 1,600 languages, 1B–8B models rival 70B LLMs — kalyan_kpl · 2026-10-28
- Uncensored LFM3-1.7B GGUF weights released on Hugging Face — kalyan_kpl · 2026-10-28
- Token demand on OpenRouter grew 27x to 165T weekly, open models now serve 72% — FinanceYF5 · 2026-10-28
- Tesla Optimus 'robot university' leak: 40-camera mocap suits, 24/7 data ops for months — Scobleizer · 2026-10-28
- Letta's FrontierMemory leaderboard: Letta Dream tops at 73.3%, beating OpenAI and Claude agents — afurgs · 2026-10-28