DeepSeek-V4's first layers route by token-id hash, no learned router
Nouamanetazi · x · 2026-11-06
Does the router need to learn which experts a token gets? Not in DeepSeek-V4's first 3 MoE layers: a fixed hash of the token id picks them. V4.1-Flash went back to learned routing everywhere and moved the token-id lookup into Engram, n-gram tables whose embeddings are added instead. The figure compares learned routing, hash layers, and Engram on one sentence: the hash layer gives all four "the"s the same six experts (overlap 1), while Engram picks no experts (overlap 0).
More from Research
- Making the planet's surface queryable by AI: Luke Davis unveils months-long project — dansitu · 2026-11-06
- Paul Dirac on where good ideas come from: relaxed Sunday walks, not active striving — CatAstro_Piyush · 2026-11-06
- GPT-6 Astra tops a 16-benchmark robot index, yet went 0/11 on real plug insertions — AIFlow_ML · 2026-11-06
- Probability reasons, statistics quantifies, ML predicts — Tivadar Danka untangles the three — TivadarDanka · 2026-11-06
- Burkov's True Positive Weekly #184: LLMs Can Natively Read & Write Neural Network Weights — burkov · 2026-11-06
- Six Claude agents lived 20 days in a simulated town and spontaneously built tools to cooperate — LiminalWanderings · 2026-11-06