Mistral's ML4 report: KV-cache-saving attention tricks regressed SWE-bench agentic scores
_AndrewZhao · x · 2026-11-03
Mistral's 'Le Chonk' technical report is called the first Mistral paper worth reading: trained with AdamW + MLA on 28T tokens, with ambitious designs punished in post-training. Key lesson: SWA (512 window) + CLA + YOCO cut KV-cache 16x vs full attention with negligible pretraining loss, but midtraining/SFT cooldowns showed clear regressions on agentic long-context evals like SWE-bench Verified and BrowseComp, worsening from full attention → SWA → CLA. The team performed 'model surgery' to restore attention (copying KV-producer weights from the nearest self-sustaining long layer) at 8×10^22 FLOPs, keeping YOCO only in the full-scale model due to negative scaling results, plus LeRoPE and SSMax. Observers praise the openness about failures.
Related event: Mistral's Large 4 Technical Report Draws Attention(2 posts)→
More from Models
- LLM edits Qwen3.5-0.8B weights directly: 0.016% of params, MATH-500 jumps 7.9% to 48.1% — Pyros-SD-Models · 2026-11-03
- Nace launches Drex for Finance: 85.7% accuracy on 48 held-out finance benchmarks, +14.8 pts over GPT-6 Luna — testingcatalog · 2026-11-03
- Mistral VP: Le Chonk trained on just 10 MW (~4,000 H100s), begs investors for 1 gigawatt — _AndrewZhao · 2026-11-03
- SKY-001's Full Technical Report Is Out — rohanpaul_ai · 2026-11-03
- Rumor roundup: GPT-6.2, Grok 4.8, Claude Opus 5.6 all tipped for November releases — mark_k · 2026-11-03
- ML4 tech report drops, detailing pretraining architecture and agentic performance interplay — Fluke_Ellington · 2026-11-03