Architecture breakdown: wider and deeper than ML3, top-4/128-expert MoE, MLA keeps KV cache tiny

eliebakouch · x · 2026-10-30

eliebakouch shares an addendum to his architecture analysis of a new model release:

• The model is both much wider and much deeper than the smaller ML3, with comparison plots against other models.
• On KV cache size there's no compression except MLA—DSV4.1 (DeepSeek V4.1) sits impressively low versus other models—while FLOPs remain relatively costly (plots normalized).
• The MoE is near-standard but unconventional: instead of the usual top-8 over 256 experts, they chose top-4 over 128 experts, likely for stability and training efficiency.

Verdict: a very cool release showing the team understood the dynamics and tradeoffs, landing on a slightly "frankenstein" KV/indexer sharing pattern that's quite efficient.

Original post →

More from Models

• Rumor: Alongside Google's Carbon, OpenAI's Internal Model "Bel" Is Coming — bindureddy · 2026-10-30
• Claude's Multi-Account Logins Enable Guilt-by-Association Bans — and More Profit — sujingshen · 2026-10-30
• Investment Arena pits 72 AI traders with $120K paper money each in live battle royale — my_cat_can_code · 2026-10-30
• NVIDIA open-sources Nemotron 3.5 Embed, Rerank and Parse models to cut agent inference token waste — NVIDIAAI · 2026-10-30
• Google AI Mode denies email access in one answer while citing emails in others — gaganghotra_ · 2026-10-30
• Qwen 3.8 Flash Next one-shots a living 3D island village in three.js at 60fps — jesdga95 · 2026-10-30

Models channel →