DeepSeek V4's fully compressed layers might bottleneck long-context performance
bookwormengr · x · 2026-07-31
The author notes that despite recent upgrades, DeepSeek V4-Flash underperforms in long-context reasoning compared to MiniMax-2.7 and Kimi K3, with even V4 Pro falling short.
The speculation is that DeepSeek V4's lack of a single full attention layer is the culprit. Its architecture consists of SWA, CSA, and HCA layers—all of which compress along the sequence dimension and are lossy by definition. This yields an extremely low KV cache size but sacrifices long-context performance.
In contrast, Kimi K3 retains 24 MLA full attention layers (25%), offering a winning compromise between cache size and long-context capabilities.
More from Models
- Why are sub-15B open models so rare now? Labs seem to have settled on 24-27B minimum — Hot_Example_4456 · 2026-10-16
- User spots model openly generating "ideal responses" to train other models — liminal_bardo · 2026-10-16
- Reflection's 501B MoE Beam hits 463 tok/s on 8x GB200 with NVFP4 at day 0 — zhyncs42 · 2026-10-16
- Gemma 4 31B QAT on one 24GB card: 46 tok/s plain, ~80 with draft model on RTX 4090 — Subject_Poet_3861 · 2026-10-16
- JevBench v1.7.46 leaderboard: new d1-ultra tops 32 API systems at 76.1 composite — airesearch12 · 2026-10-16
- Team of 5 mulls dropping ChatGPT Pro for Business as OpenAI quota parity stays unclear — bettercallfool · 2026-10-16