DeepSeek V4.1 Flash notes: how obsessing over KV cache compression yields a hyper-efficient frontier model
nrehiew_ · x · 2026-09-11
nrehiew publishes a deep technical thread on DeepSeek V4.1 Flash, themed "how obsessing over KV cache compression gets you a hyper-efficient frontier model". Coverage includes:
- Multimodal pretraining on 45T image-text tokens, custom image encoder, modality-level load balancing
- YOCO-style 20-layer encoder + 20-layer decoder halving KV cache
- Upgraded compressed attention (CSA): three KV-reuse variants with a sparse indexer that is itself sparse
- Head-wise Muon optimizer, 196B-param Engram with Sinkhorn Balancing, QAT-based FP4 KV cache
- Infra: Siglip-style vision encoder, shadow indexers, optimizer sharding across Engram tables
- Fully data-centric post-training: synthetic multi-agent trajectories and checkpoint merging
A high-quality first-hand explainer of DeepSeek's latest engineering trade-offs.
More from Models
- Novel Reasoning Effort Control Scheme Analyzed: Graded GRPO Training — stochasticchasm · 2026-09-11
- Forcing models to always max effort is like humans evolving on Adderall, researcher argues — voooooogel · 2026-09-11
- Pushing models to always show 'maximum effort' drags along its corollaries, dev argues — voooooogel · 2026-09-11
- Claude is the distillation target of choice because agentic RL seed data is scarce — teortaxesTex · 2026-09-11
- Why Chinese labs distill from Anthropic: Claude's agent data is the scarce training signal — teortaxesTex · 2026-09-11
- ApprenticeBench: closed model scores 72% vs open Kimi K3 at 18% on real jobs — ysu_nlp · 2026-09-11