DeepSeek v4 insider: mHC residual streams may converge — why iHC is the safer pick

teortaxesTex · x · 2026-10-28

teortaxesTex amplifies an analysis by former DeepSeek researcher khazzz1c (Chonghan Liu) on mHC (multi-residual-stream mixing), plus a Grok translate failure that dropped the arXiv link.

Core claim: mHC constrains mixing of residual streams via doubly stochastic matrices, fixing numerical stability — but Numerical Stability ≠ Information Preservation. Differences between streams may decay with depth, converging toward the same mean on the pure mixing path.
Empirical match: this aligns with experiments run on the DeepSeek v4 model (mHC ultimately converging toward a single-HC case), one reason the author recommends iHC instead.
Trade-off: preserving stream differences requires near-permutation mixing matrices, which brings Vertex Viscosity and slower Sinkhorn convergence — a trade among stability, information preservation, and optimization efficiency.
Caveat: full mHC doesn't necessarily degenerate into one stream since each layer injects new information; the open question is how deep added residual streams can retain independent information.
Attached analysis of DeepSeek-V4-Flash's 43 layers: layers 0-21 need residual mixing (identity swap raises perplexity 41%), while layers 22-42 learned mixers are already near-identity — turning off cross-stream mixing late is nearly free.

Original post →

DeepSeek v4 insider: mHC residual streams may converge — why iHC is the safer pick

More from Models