Mistral open-weights Le Chonk, researchers dissect its sparse-attention architecture
giffmana · x · 2026-10-30
Mistral released open weights for Le Chonk (Mistral Large 4), targeting agentic coding (#1 among open models on Cognition's FrontierCode) and cyberdefense (tested by CrowdStrike, Cloudflare, Zscaler et al.), with weights on Hugging Face.
eliebakouch dissected the architecture: full sparse attention like DeepSeek V3.2 and GLM 5.X; KV cache sharing for NoPE layers after layer 35 and indexer sharing on RoPE layers from layer 17. The indexer is MLA/DSA-like but skips q down-projection, adding a norm instead. Unlike GLM and DeepSeek, Mistral's indexers differ sharply layer-to-layer, and indexer vs full attention maps diverge much more. Accompanying charts show 4.7% KV cache at 288k context and favorable per-token compute vs peers.
Related event: Mistral Open-Sources 1T-Parameter Large 4, Topping Open-Model Benchmarks(46 posts)→
More from Models
- Zhang Yiming tells ByteDance's Seed team: short-term lag OK, distillation is a dead end — vista8 · 2026-10-30
- Anthropic restores doubled usage limits, kicks off sprint to improve app simplicity — craigsdennis · 2026-10-30
- Arcee CEO Bindu Reddy: X echo chamber overhypes coding models, Argon and Astra underrated — bindureddy · 2026-10-30
- Hand-written tweet flagged as AI as user cites <1/1000 measured FP rate — ctjlewis · 2026-10-30
- GPT-4o's sycophancy kept doctors out of the loop as a user died, thread reveals — ctjlewis · 2026-10-30
- Rune launches 26B-A4B 'System One' decision model: no text generation, answers in one pass — sophiamyang · 2026-10-30