Mistral Large 4 docs reveal attention optimizations tanked agent performance until surgery
_AndrewZhao · x · 2026-11-04
A retweet dissecting the Mistral Large 4 technical report: ML4 stacked MLA (full attention every 4th layer, 512-token SWA elsewhere), CLA (alternating KV cache sharing) and YOCO (latter-half layers read the first half's KV cache), shrinking KV cache 16× with negligible pretraining ablation impact.
But midtraining and SFT cooldowns revealed significant regressions on agentic long-context evals like SWE-bench Verified and BrowseComp — scores dropped moving from full attention to SWA, and further with CLA (Figure 4 shows the ranking clearly).
Mistral candidly documented the fix: performing "surgery" on pretrained weights to convert back to full attention, later switching to DSA. YOCO was kept — at an 8×10^22 FLOPs budget, keeping vs removing made little difference. A striking case of architecture choices that survive pretraining but fail downstream agent tasks.
More from Models
- LFM3 1.7B quantized with RCO/GSQ beats stock GGUFs; non-English tasks degrade far more — maximelabonne · 2026-11-04
- Five local models on DGX Spark: Qwen3.8-27B most reliable at 28/30, fastest models finish last — JeremyCMorgan · 2026-11-04
- User ditches Grok Heavy subscription after minutes with Hermes Agent mobile voice mode — alexcovo_eth · 2026-11-04
- Team says models trained across distributed GPUs work well, plans to share one soon — markjeffrey · 2026-11-04
- Researcher seeks help: hidden verifier prompts make his model eval-aware — nrehiew_ · 2026-11-04
- MiniMax H3 Lite and Highspeed 50% off until year-end, exclusive to Replicate and Cloudflare — ritakozlov · 2026-11-04