DeepSeek-V4.1-Flash architecture dissected: asymmetric causal encoder-decoder, CSA2 attention, Engram memory

teortaxesTex · x · 2026-09-11

A detailed public breakdown of DeepSeek-V4.1-Flash, the novel architecture from DeepSeek's latest technical report:

Asymmetric causal encoder-decoder (CED): a 40-layer network split into 20-layer causal encoder and decoder (not BERT/T5-style) — generated tokens still pass through all 40 layers, but prompts only need the 20 encoder layers plus a 128-token decoder replay, since decoder global KV is projected from the encoder's final hidden state. 8B params activate per token in prefill vs 16B in decode, nearly halving prefill compute.
CSA2 attention: first two encoder layers use sliding window attention; the rest use Compressed Sparse Attention 2 with Full/Reindex/Reuse modes that reuse main KV and Top-K indices across layers (encoder: SWA, SWA, then [Full + 5 Reuse]×3; decoder: Full + 3 Reuse, then [Reindex + 3 Reuse]×4). V4.1 drops V4's CSA–HCA hybrid for pure CSA2, cutting cache storage and indexer compute.
Other components: standard DeepSeekMoE FFNs, the Engram conditional memory module decoupling memorization from computation, Single-Pass mHC, DSpark, and a Hierarchical Sparse Indexer.

Overall it's a long-context efficiency architecture centered on saving KV cache and prefill compute — the most detailed public dissection of DeepSeek's new architecture so far.

Original post →

DeepSeek-V4.1-Flash architecture dissected: asymmetric causal encoder-decoder, CSA2 attention, Engram memory

More from Models