IFM launches K2-Horizon-36B-A4B: 4B active params match models 20x its size
rupspace · x · 2026-09-22
IFM has released K2-Horizon-36B-A4B, a model built on its new MoVA (Mixture-of-Value Attention) architecture, now on Hugging Face with code and technical report coming soon.
- Score: 25 on the Artificial Analysis Intelligence Index, matching models with over 20× its total parameters while using only 4B active parameters per token.
- Architecture: brings MoE-style sparsity into the value vectors of multi-head attention, opening a second axis for scaling sparsity in LLMs beyond FFN-based MoE.
- Engineering wins: compatible with flash attention, GQA, and sparse attention; no extra KV-cache cost versus standard GQA.
More from Models
- OpenAI's five biggest failures laid out: pretraining decline, alignment fears, enterprise lost to Claude — teortaxesTex · 2026-11-12
- Fable 5.2 Hits 41.3% on Remote Labor Index, Up From 2.5% a Year Ago — alliekmiller · 2026-11-12
- Beaker automates MIT's weight-space programming: one-neuron edit lifts MATH-500 from 7% to 37% for under $4 — VictorSanh · 2026-11-12
- Researcher says Anthropic dodged questions on Mythos false positive rate; 37,966 findings, 2 exploited — OwariDa · 2026-11-12
- Grok spotted building meeting "Listen to Conversation" feature with live transcript — nima_owji · 2026-11-12
- MoonMath ships uncensored 763B DeepSeek-V4.1 via closed-form weight editing, claims zero capability loss — udmrzn · 2026-11-12