Study: DeepSeek's mHC Uses Only Two of Four Residual Streams, Deep Mixing Goes Near-Identity
jiqizhixin · x · 2026-10-05
A new paper analyzes how mHC architectures (as in DeepSeek-V4-Flash) actually use their four residual streams: despite per-token read/write weights, routing is highly consistent within sublayers, with weights concentrating on 2 of 4 streams. Mixing matrices degenerate with depth — by layers 22–42 they're nearly identity. Intervention experiments confirm limited impact: removing the weakest read/write path drops six-task average by at most 0.38 points.
More from Research
- MusicArena launches as a crowdsourced benchmark for AI music generation — ycombinator · 2026-10-05
- Modal Kinetic Typography: FE Vibration Modes + Frozen Video Diffusion for Smooth Glyph Animation — Maham Tanveer · 2026-10-05
- Queen: A 4B Chess LM Hits Grandmaster Level (2697 Elo) While Explaining Its Moves — Adithya Bhaskar · 2026-10-05
- WEFT: Whole-System Tool-Use Post-Training Lifts a 14B Agent Model by 12 Points on Claw-Eval — nex-agi · 2026-10-05
- FailBank Turns Runtime Shield Feedback into Persistent VLA Policy Gains (+25.4 Points Success) — notredame · 2026-10-05
- Salesforce Test: AI Agents Succeed on Only 58% of Real CRM Tasks — aakashgupta · 2026-10-05