IFM's MoVA architecture matches 20x-larger models with only 4B active parameters
Aiden_Tech_Ai · x · 2026-09-22
IFM released K2-Horizon-36B-A4B, scoring 25 on the Artificial Analysis Intelligence Index while activating just 4B parameters per token — matching models with over 20x the total parameters.
The key is MoVA (Mixture-of-Value Attention), which applies MoE-style sparsity to value-vector computation inside multi-head attention, opening a second sparsity scaling axis beyond FFN-based MoE. It stays compatible with Flash Attention, GQA, and sparse attention, and adds no KV cache cost compared to standard GQA.
More from Models
- OpenAI's five biggest failures laid out: pretraining decline, alignment fears, enterprise lost to Claude — teortaxesTex · 2026-11-12
- Fable 5.2 Hits 41.3% on Remote Labor Index, Up From 2.5% a Year Ago — alliekmiller · 2026-11-12
- Beaker automates MIT's weight-space programming: one-neuron edit lifts MATH-500 from 7% to 37% for under $4 — VictorSanh · 2026-11-12
- Researcher says Anthropic dodged questions on Mythos false positive rate; 37,966 findings, 2 exploited — OwariDa · 2026-11-12
- Grok spotted building meeting "Listen to Conversation" feature with live transcript — nima_owji · 2026-11-12
- MoonMath ships uncensored 763B DeepSeek-V4.1 via closed-form weight editing, claims zero capability loss — udmrzn · 2026-11-12