xHC: A New Architecture Breaking Transformer Residual Stream Scaling Bottlenecks
rednote-hilab · hf · 2026-07-20
Existing hyper-connection methods are typically limited when scaling beyond 4 parallel residual streams due to diminishing returns and soaring training costs. The study notes this is mainly constrained by insufficient write-back information and the cubic growth of residual mixing costs.
To address this, the research introduces xHC (Expanded Hyper-Connections), the first method to achieve effective scaling for N>4. Its core designs include:
- Temporal Feature Enhancement: Provides richer write-back information.
- Sparse Residual Stream Architecture: Updates only k=4 out of N=16 streams while maintaining dense access to the global state.
Experiments on 18B and 28B MoE models show that xHC delivers significant downstream performance improvements with moderate training overhead. Furthermore, the proposed xHC-Flash technique effectively controls memory bandwidth traffic, reducing overhead to a level comparable to mHC with N=4, making large-scale residual stream scaling genuinely practical.
Related event: xHC Architecture Breaks Transformer Residual Stream Limits(2 posts)→
More from Research
- Stanford Data-Centric AI Workshop reopens site, extends poster deadline — jure · 2026-10-26
- Hugging Face wraps 6-month Training Agents series with open recordings and code — SergioPaniego · 2026-10-26
- 'Vibe verification': ~60% of AI-written formal specs prove the wrong property — basedjensen · 2026-10-26
- Ito: a browser that detects browsing intent and auto-organizes tabs into Flows — andrewkuznet · 2026-10-26
- Cornell study says brains process language as linear sequences, challenging Chomsky's hierarchy — anirbanbandyo · 2026-10-26
- Hi3D V1.0 open-sourced with 2M training assets and 90% slicing success rate — kimmonismus · 2026-10-26