xHC: New Paper on Expanding Residual Streams
burny_tech · x · 2026-07-18
The paper xHC: Expanded Hyper-Connections proposes expanding the Transformer's residual stream into N parallel streams, utilizing sparse updates and convolution-style writeback to mitigate information loss and excessive writeback costs after expansion.
The authors report that xHC delivers consistent downstream performance improvements on 18B and 28B MoE models; on the 18B MoE, the average downstream score is 4.0 points higher than mHC. They also note that to reach the same loss, the vanilla baseline requires roughly 1.50× the compute and mHC requires 1.19×, demonstrating that xHC further enhances the feasibility of large-scale residual-stream expansion.
More from Research
- TMLR paper: enforced thinking doesn't reliably make LLM agents disclose key info to users — SharonYixuanLi · 2026-10-27
- Fern architecture claims 10M-token context with predictions still improving — AccBalanced · 2026-10-27
- Diamandis: AI Cancer Navigator and $100 Genome Sequencing Close the Care Gap — PeterDiamandis · 2026-10-27
- EMNLP Oral: training-free Nexus Sampling cuts KV-cache memory 10x; OpenResearcher tops BrowseComp-Plus at 54.8% — yuz9yuz · 2026-10-27
- ICML 2026 position paper: LLM 'thinking traces' are a harmful anthropomorphism — rao2z · 2026-10-27
- Forking gap follows a power law in context frequency; masking rare n-grams shrinks it — ZimingLiu11 · 2026-10-27