Leaked DeepSeek V4.1 Flash Tech Report Reveals KV Cache-Obsessed Flagship

A technical report for DeepSeek V4.1 Flash leaked online; @burkov cautions the release is not officially confirmed by DeepSeek. The report describes a flagship model built around KV cache compression: @stochasticchasm notes startling benchmark numbers, a 4x smaller KV cache than DSV4-Flash, and more stable training; @nrehiew published a long deep-dive series, with complementary analyses from Mhr1036 (via @demianai) and arjunkocher (via @teortaxesTex). Community consensus: systematic innovation across architecture, algorithms, training, and inference simultaneously cuts compute and memory costs for long context.

Confirmed

• 552B-parameter MoE backbone with a 1M-token context window; Mhr1036 notes the "paying to keep context" second cost of long context, which this design compresses on both compute and memory; @stochasticchasm reports a 4x smaller KV cache vs DSV4-Flash
• Core architecture is an asymmetric causal encoder-decoder (CED): 40 layers split into 20 encoder + 20 decoder; @nrehiew explains lower layers generate KV cache shared upward via per-layer projections — a YOCO-style decoder-decoder halving KV cache; arjunkocher confirms the asymmetric design and CSA2 attention; @stochasticchasm corrects: the decoder only periodically reads final encoder states, not standard cross-attention
• Per Chris Alexiuk (via @altryne), prefill/decode separation is "baked" into the model weights, not just the serving stack
• Global attention is upgraded compressed attention: three KV-reuse variants all run a sparse indexer over a small KV, then concatenate selected tokens with windowed KV; the indexer itself is hierarchical (full-mode first-stage candidate selection)
• Head-wise Muon optimizer (including vision), a 196B-parameter Engram component with Sinkhorn balancing; sparse attention trained from scratch without warmup; @stochasticchasm notes no input/output optimizer parameters, weight decay only on the LM head, and suspects Muon is becoming the training default
• KV cache is quantization-aware trained (QAT), explaining strong fp4 KV cache performance
• Multimodal: 45T image-text tokens, in-house SigLIP-style encoder, modality-level load balancing (reminiscent of Chameleon per @nrehiew)
• RL infrastructure (@nrehiew): dispatch strategies eliminating tail stalls, router replay from old checkpoints, off-policy controls for short completions, 40+ teacher OPDs; efficiency rated "Sol/Opus-level" with "insane" benchmark numbers
• Sandbox & data (@nrehiew): in-house orchestrator, 2500 concurrent sandboxes per node per VM within NUMA domains; agent trajectory data explicitly inspired by real partner usage patterns, synthesized by multiple agents, with checkpoint merging
• Inference (@nrehiew): heavily fused kernels (15 for prefill, 11 for decode); SWA cache kept per user/assistant message (minute-level); first model @nrehiew has seen with a numeric reasoning-effort parameter directly affecting length penalty
• Training infra: parallel all-gather of one modality's features while another computes, context parallelism for long multi-image sequences, dedicated sparse-indexer clusters
• Design almost solely targets inference efficiency: friendly to PD disagg with good utilization at large prefill batches; some commenters find v4's CSA/HCA dual compression-frequency design under-explained vs pure CSA2
• @stochasticchasm observes DeepSeek appears to break its "V<N> = new pretrain" naming convention; some suggest this generation is a simplified V4

Unconfirmed

• @burkov stresses the report is not officially confirmed by DeepSeek
• @stochasticchasm's speculations linking the leaked architecture to rumored qwen-3.8-flash-next (51B n-gram parameters) and mechanisms like globally shared KV + per-layer SWA are personal conjecture

Why it matters

The community takeaway is that obsession with KV cache compression produced an exceptionally efficient flagship. As 1M-token contexts and long-horizon agent workloads drive up KV cache read/write and memory costs, V4.1 Flash demonstrates a systemic cost-reduction path — from architecture (asymmetric CED, KV reuse, baked-in PD separation) and algorithms (hierarchical sparse indexer, compressed attention, KV QAT) to training (Muon, Sinkhorn, router replay) and deployment (15/11 fused kernels, message-level SWA cache, numeric reasoning effort) — a valuable blueprint for long-context multimodal frontier models.

2026-09-11 ~ 2026-09-11 · 44 related posts

Full story(18 episodes)→

• Episode 1: DeepSeek V4.1 Flash Opens Limited Internal Beta with New Architecture and Native Multimodality(2026-09-08, 18 posts)
• Episode 2: DeepSeek V4.1 Flash Tested: Blazing 350 Tokens/s but Still Experimental(2026-09-08, 2 posts)
• Episode 3: DeepSeek cuts V4-Flash API prices with new peak/off-peak billing from Sept 10(2026-09-08, 6 posts)
• Episode 4: DeepSeek V4.1 Flash Nears Frontier Models at a Fraction of the Cost, Independent Tests Show(2026-09-09, 15 posts)
• Episode 5: DeepSeek Releases Open-Source V4.1 Flash, Beating Its Own Flagship at a Fraction of the Cost(2026-09-09, 57 posts)
• Episode 6: DeepSeek V4.1 Flash Leaks: 552B Asymmetric MoE Claimed to Rival GPT-5.6 Sol(2026-09-10, 21 posts)
• Episode 7: DeepSeek V4.1 Flash Benchmark Results Leak, Sparking Community Discussion(2026-09-10, 2 posts)
• Episode 8: DeepSeek Shrinks Per-Token KV Cache 54x in Nine Months to Sub-KB Levels(2026-09-10, 6 posts)
• Episode 9: Leaked DeepSeek-V4.1-Flash Touted as New Open-Source King(2026-09-10, 3 posts)
• Episode 10: Bug Hunt Bench Ranks Frontier Coding Models on 105 Real Bugs(2026-09-10, 13 posts)
• Episode 11: DeepSeek V4.1 Flash Confirmed to Center on YOCO Architecture(2026-09-10, 7 posts)
• Episode 12: Leaked DeepSeek V4.1 Flash Tech Report Reveals KV Cache-Obsessed Flagship(2026-09-11, 44 posts)
• Episode 13: DeepSeek V4.1 Flash tops Vals open-source index at $0.30 per run(2026-09-11, 2 posts)
• Episode 14: DeepSeek launches V4.1-Flash: 552B backbone, 1M-token context(2026-09-11, 5 posts)
• Episode 15: DeepSeek V4.1 Flash scores 40 in AA index despite rising hallucinations(2026-09-11, 2 posts)
• Episode 16: Inside KV-Cache Sharing: How DeepSeek CED and GLM 5.2 Differ(2026-09-11, 4 posts)
• Episode 17: DeepSeek 4.1 Flash Shows Big Gains in Kernel Engineering and Cost(2026-09-11, 2 posts)
• Episode 18: SGLang delivers Day 0 support for DeepSeek V4.1 Flash, hitting 873 tok/s(2026-09-12, 2 posts)

• DeepSeek-V4.1-Flash architecture dissected: asymmetric causal encoder-decoder, CSA2 attention, Engram memory — teortaxesTex · 2026-09-11
• Blogger flags new model's standout tech report: high benchmarks and 4x smaller KV cache vs dsv4-flash — stochasticchasm · 2026-09-11
• [source] DeepSeek's New Model: 4x Smaller KV Cache Than DSV4-Flash and More Stable Training — stochasticchasm · 2026-09-11
• DeepSeek appears to break its V<N> naming convention, new architecture said to train more stably — stochasticchasm · 2026-09-11
• New attention design called inference-informed: cheap prefill, small KV cache — stochasticchasm · 2026-09-11
• Inference efficiency is now the sole driver of architecture changes, observer notes — stochasticchasm · 2026-09-11
• Pure CSA2 beats v4's alternating CSA/HCA compression, argues arch observer — stochasticchasm · 2026-09-11
• Straight-up FP4 KV cache with no degradation: inference engineers take note — stochasticchasm · 2026-09-11
• FP4 KV cache with no degradation sparks debate on aggressive inference compression — stochasticchasm · 2026-09-11
• New model adds engram and simplified mHC; community digs into KV cache tradeoffs — stochasticchasm · 2026-09-11
• Sparse attention without dense warmup signals midtraining absorption, analyst argues — stochasticchasm · 2026-09-11
• Sparse Attention Without Dense Warmup Signals Long-Context Training Breakthrough, Analyst Argues — stochasticchasm · 2026-09-11
• Decoder doesn't cross-attend to encoder KVs — just periodically reads final encoder states — stochasticchasm · 2026-09-11
• Architecture deep-dive: decoder periodically reads final encoder states instead of cross-attention — stochasticchasm · 2026-09-11
• Why Only Two Layers Use Sliding Window Attention? Architecture Detail Sparks Debate — stochasticchasm · 2026-09-11
• Leaked Qwen next-gen model shows record n-gram params, first two layers SWA-only — stochasticchasm · 2026-09-11
• Tiny KV Cache via Shared Global KV Plus Per-Layer SWA? New Architecture Speculation — stochasticchasm · 2026-09-11
• Engram section analysis: prime-sized tables, 4-grams, and fp8 lookup tables — stochasticchasm · 2026-09-11
• MTP dropped in new model's tech report: fp8 lookup tables and prime table sizes noted — stochasticchasm · 2026-09-11
• Model reportedly uses QAT for KV cache, explaining strong fp4 performance while dropping MTP — stochasticchasm · 2026-09-11
• KV cache gets QAT too: why this model beats others at fp4 KV cache — stochasticchasm · 2026-09-11
• Another Model Shifts to Muon Optimizer as It Emerges as the Training Default — stochasticchasm · 2026-09-11
• Muon optimizer becomes the default as new models drop Adam entirely — stochasticchasm · 2026-09-11
• DeepSeek V4.1 Flash Architecture: 552B MoE with Asymmetric 8B Read / 16B Decode Compute — demian_ai · 2026-09-11
• No Input/Output Optimized Params, and No Weight Decay on the LM Head — stochasticchasm · 2026-09-11
• [source] DeepSeek V4.1 Flash notes: how obsessing over KV cache compression yields a hyper-efficient frontier model — nrehiew_ · 2026-09-11
• DeepSeek V4.1 Flash multimodal: 45T image-text tokens and modality-level load balancing — nrehiew_ · 2026-09-11
• DeepSeek V4.1 Flash halves KV cache with YOCO-style decoder-decoder design — nrehiew_ · 2026-09-11
• DeepSeek V4.1 Flash's global attention: compressed attention with three KV-reuse variants — nrehiew_ · 2026-09-11
• DeepSeek V4.1 Flash: even the sparse indexer is itself sparse, with hierarchical candidate selection — nrehiew_ · 2026-09-11
• DeepSeek V4.1 Flash uses Muon optimizer and Sinkhorn balancing for 196B-param Engram — nrehiew_ · 2026-09-11
• DeepSeek V4.1 Flash infra: Siglip-style vision encoder and shadow indexer workers — nrehiew_ · 2026-09-11
• DeepSeek V4.1 Flash post-training is fully data-centric: synthetic multi-agent trajectories plus checkpoint merging — nrehiew_ · 2026-09-11
• Only 15 Fused Kernels in Prefill: V4.1's Inference Stack and Minute-Scale SWA Cache — nrehiew_ · 2026-09-11
• 2,500 Concurrent Sandboxes Per Node, Cross-Scaffold Checkpoint Merging Yield Free Gains — nrehiew_ · 2026-09-11
• V4.1 Is the First Model With a Numerical Reasoning Effort Parameter Tied to Length Penalty — nrehiew_ · 2026-09-11
• RL Training Infra Breakdown: Router Replay, Off-Policy Controls, Full-Vocab OPD on 40+ Teachers — nrehiew_ · 2026-09-11
• New Model Hits Opus-Level Benchmarks at Wild Efficiency, RL Infra Details Emerge — nrehiew_ · 2026-09-11
• V4.1's Reasoning Curve Isn't Linear; Agent Teams Explicitly RL-Trained to Collaborate — nrehiew_ · 2026-09-11
• DeepSeek's new tech report: 890 bytes/token KV, inference-first architecture dissected — nrehiew_ · 2026-09-11
• Novel Reasoning Effort Control Scheme Analyzed: Graded GRPO Training — stochasticchasm · 2026-09-11
• Unverified DeepSeek-V4.1-Flash report: 552B MoE slashing KV cache for million-token agent workloads — burkov · 2026-09-11
• [source] DeepSeek V4.1 Flash reportedly bakes prefill/decode disaggregation into the model weights — altryne · 2026-09-11

1 near-duplicate retellings: altryne