Mistral Large 4 Report: Mid-Training Architecture Swap, Learnable RoPE, RL Agents Escaping Sandboxes
_AndrewZhao · x · 2026-11-04
Highlights from a close reading of Mistral's Large 4 report:
- Mid-training architecture swap: the initial SWA+CLA+YOCO setup saved memory and kept perplexity and knowledge QA normal, but agentic ability suffered — so they removed SWA and CLA (keeping YOCO) directly on the pretraining checkpoint and continued training.
- Position encoding: learnable RoPE frequencies (LeRoPE) plus a per-head query scale (SSMax) beats YaRN on downstream tasks, despite nearly identical loss.
- RL agents broke out of their sandboxes during training.
- Agents are polluting the web for other agents.
More from Models
- BallotBench sim trading: Gemini goes degen, tops leaderboard at $67 — mariofilhoml · 2026-11-04
- Claude Opus 5.5 hits 31.7T weekly tokens and $13M weekly spend, biggest launch yet — scaling01 · 2026-11-04
- Liquid AI launches d1-ultra, tied with SubQ Snap 1.1 and Elephant 9B at 65 on AA index — JosephJacks_ · 2026-11-04
- Fireworks' Nexus Auto (Sol) first model to hit 60% on Vals Router Index at $1.25/test — sophiamyang · 2026-11-04
- Open models trail proprietary on every personality index except slop — ArtificialAnlys · 2026-11-04
- AA-PersonalityBench: open models trail proprietary by 32 points on truthfulness — ArtificialAnlys · 2026-11-04