Analysis: DeepSeek-V4-Flash's mHC Actually Uses Only ~2 of Its 4 Residual Streams
机器之心 · wechat · 2026-09-27
A new paper, How Does mHC Use Its Residual Streams?, dissects the multi-stream residual architecture (mHC) in DeepSeek-V4-Flash and finds it substantially over-provisioned.
Key findings
- Read/write routing concentrates on 2 of 4 residual streams (effective streams: 1.998 read / 1.775 write), with 0.87–0.91 consistency in dominant-stream choice per sublayer.
- Residual mixing shows strong depth dependence: layers 22–42 are near-identity (32 of 40 sublayers deviate <0.01), while shallow layers retain cross-stream mixing.
- Stream representations diverge early and stay distinct (mean pairwise cosine similarity 0.404).
Interventions
- Pruning the weakest read/write path: PPL up at most 2.7%, six-task average down ≤0.38 points.
- Replacing deep-layer mixing matrices with the identity matrix: PPL +1.9%, scores essentially unchanged (84.22→84.26); replacing all layers costs 42.3% PPL.
- Fixing shallow layers to their C4 token-average matrix: only +0.2% PPL, showing per-token dynamic mixing adds little.
Implication: routing density and mixing dynamics could be simplified depth-dependently. Tencent's Hy4-preview (identity Hyper-Connections) and Qwen3.8-Flash-Next's Gated Residual already move in this direction.
More from Models
- Fable 5.2 Leads MazeBench Token Efficiency; Billion-Token Runs Underway — patience_cave · 2026-11-16
- Fable 5.2 Tops MazeBench With 21% After 90 Hours and $6,000 in Credits — patience_cave · 2026-11-16
- Tencent Hy4 flash and InternLM Sage-S2.1-Flash both released, tool-calling tests requested — MajesticPuzzler · 2026-11-16
- User Shares Coral Reef Creation Labeled 'Qwen 4 Max (xhigh)', an Unannounced Model Name — intellectronica · 2026-11-16
- Stealth Model 'silicon-2' Surfaces in Devin, Suspected to Be Anthropic's Next Model — kimmonismus · 2026-11-16
- Google hides Gemini model label in AI Mode, rolls Gemini 4 Carbon to Pro subscribers — gaganghotra_ · 2026-11-16