DeepSeek cites 2024 YoCo paper as inspiration behind its CED transformer blocks

jm_alexia · x · 2026-09-10

DeepSeek reportedly cited the 2024 YoCo (You Only Cache Once) paper as the main inspiration for its Causal-Encoder Decoder (CED) transformer blocks.

YoCo's decoder-decoder design: the model splits into two stacked halves — a Self-Decoder (bottom half) processing input with efficient causal attention to produce a single shared global KV cache, and a Cross-Decoder (top half) whose layers cross-attend to that one global KV cache instead of computing their own KV pairs.
Key benefit: expensive global KV pairs are computed once and reused across all upper layers, cutting memory and inference cost.
The author calls this the "secret sauce" behind DeepSeek-4.1 and marvels at the unexpected revival of encoder-decoder architectures.

More from Models