DeepSeek's V4.1 Flash reworks the Transformer again: asymmetric encoder-decoder and CSA2
evijit · x · 2026-09-10
An analysis (unconfirmed) of the DeepSeek V4.1 Flash paper argues the headline isn't parameters or benchmarks but that DeepSeek is changing the Transformer itself, with three major architectural shifts:
- Back to encoder-decoder, asymmetrically: 20 causal-encoder layers (8B active on input) plus 20 decoder layers (16B active on output) — reading a million tokens and generating the next token are different computations and shouldn't share one structure.
- CSA upgraded to CSA2 with cross-layer attention reuse: three modes — Full (recompute KV + Top-K index), Reindex (reuse KV, re-decide which tokens to attend), Reuse (reuse both) — yielding sequences like Full → Reuse → Reuse → Reindex, on the bet that "which tokens matter" needn't be recomputed every layer.
- Engram returns: a conditional-memory module for fixed collocations, entities and local patterns.
evijit adds that open-releasing these details raises the technical floor of the whole research ecosystem.
Related event: DeepSeek Returns to Encoder-Decoder Architecture and Open-Sources It(3 posts)→
More from Models
- DeepSeek V4.1 paper praised as a top-5 DeepSeek paper, textbook-style — teortaxesTex · 2026-09-10
- Google AI Mode now cites 72% fewer sources for logged-out users, data shows — gaganghotra_ · 2026-09-10
- DeepSeek cites 2024 YoCo paper as inspiration behind its CED transformer blocks — jm_alexia · 2026-09-10
- Research has cut LLM costs over 10x, and model architecture is the only math lever, argues thread — ChengleiSi · 2026-09-10
- DeepSeek unveils asymmetric Causal Encoder-Decoder: 552B MoE with just 8B active input params — ChengleiSi · 2026-09-10
- Switch Transformer by hand: a 13-step walkthrough of how sparse MoE works — ProfTomYeh · 2026-09-10