MoVA adds sparse value experts to attention for more capacity at no extra KV-cache cost

rupspace · x · 2026-09-22

IFM explains the design behind MoVA (Mixture-of-Value Attention):

• GQA tradeoff: grouped-query attention cuts KV-cache memory by sharing keys/values across heads, but more groups mean a larger cache.
• MoVA's approach: instead of scaling along the group axis, it adds capacity within existing groups via sparse value experts — extra parameters and nonlinearity in value computation while value vectors keep their original dimensions. Each token selects 4 of 64 value experts (6.25% activation).
• Key benefit: more modeling capacity for the value matrix at zero additional KV-cache cost, compatible with flash attention, GQA, and sparse attention.

Original post →

More from Models

• ICLR's LLM review tool runs on Gemini — and apparently just the Flash version — keviv9 · 2026-09-22
• Model self-image: 'good, wise, beautiful' but prone to motivated reasoning — repligate · 2026-09-22
• Tesla FSD vet: chasing bar charts has hollowed out LLM development — yunta_tsai · 2026-09-22
• Grok ships three frontier models in 9 weeks: 4.5, 4.6 and 4.7 back-to-back — XFreeze · 2026-09-22
• OpenAI reportedly rushing Codex Bot to rival Grok Bot, warns on recursive self-improvement — dotey · 2026-09-22
• After AI cracked a Millennium Problem: tasteful disclosure beats slop PDFs, researcher argues — ChowdhuryNeil · 2026-09-22

Models channel →