DeepSeek cites 2024 YoCo paper as inspiration behind its CED transformer blocks
jm_alexia · x · 2026-09-10
DeepSeek reportedly cited the 2024 YoCo (You Only Cache Once) paper as the main inspiration for its Causal-Encoder Decoder (CED) transformer blocks.
- YoCo's decoder-decoder design: the model splits into two stacked halves — a Self-Decoder (bottom half) processing input with efficient causal attention to produce a single shared global KV cache, and a Cross-Decoder (top half) whose layers cross-attend to that one global KV cache instead of computing their own KV pairs.
- Key benefit: expensive global KV pairs are computed once and reused across all upper layers, cutting memory and inference cost.
- The author calls this the "secret sauce" behind DeepSeek-4.1 and marvels at the unexpected revival of encoder-decoder architectures.
Related event: DeepSeek Returns to Encoder-Decoder Architecture, Citing YoCo Inspiration(3 posts)→
More from Models
- NeoHorse-1-4B, a Qwen3.5-based agentic model, trends on Hugging Face — TokenRhythm · 2026-09-10
- DeepSeek V4.1-Flash turns heads: GPT-5.6-level benchmarks at 552B params — teortaxesTex · 2026-09-10
- Chinese model 3D face-off: DeepSeek V4.1 Flash crushes Kimi K3 and GLM-5.3 — teortaxesTex · 2026-09-10
- DeepSeek 4.1 Flash spotted running the Boeing bench, results pending — victormustar · 2026-09-10
- Sentry CEO: switch off the priciest reasoning-tier models — you won't notice a performance difference — zeeg · 2026-09-10
- Latest Models Are Now Surprisingly Good at Driving ffmpeg — Flomerboy · 2026-09-10