Mistral Large 4 architecture teardown: sparse attention plus quirky KV/indexer sharing

eliebakouch · x · 2026-10-30

Overview: Mistral Large 4 is now open-weight: 64.8B total params with 8.8B active per token, 72 layers, 64 heads — much wider and deeper than Mistral 3.
Fully sparse attention: follows the DSA/MLA-style route of V3.2 and GLM 5.x; a small indexer (32 heads × 128) picks which of 2048 keys attention actually reads. KV cache sharing starts at layer 35 for NoPE layers (11% cache savings), and indexer sharing at layer 39 for RoPE layers (36% indexer FLOPs saved) — a "frankenstein" but efficient sharing pattern.
Indexer findings: indexers differ greatly across layers; GLM and DeepSeek show blockwise indexer structure, while Mistral only resembles them in late layers that start sharing KV. Indexer-derived maps differ noticeably from full attention maps, unlike DeepSeek/GLM.
Two long-context knobs: SSMax adds a per-head learned term that sharpens attention as context grows (Llama4/Muse and most Chinese models don't do this); LeRoPE = YaRN plus a learned per-layer decay term (×2.5 from layers 16–17), cutting per-token compute 45% at 128k context.
MoE: standard top-8/256-expert config was available, but they chose top-4 with 128 experts, likely for stability/training efficiency. KV cache has almost no compression beyond MLA — notably smaller than DSv4.1 — and compute is cheap. Author's verdict: a well-understood, excellent release. Congrats to Mistral.

More from Models