xHC scales Transformer residual streams beyond N=4

pmttyji · reddit · 2026-07-20

This paper proposes xHC (Expanded Hyper-Connections), a new way to scale Transformer residual streams beyond the usual width/depth axis.

Core idea

Hyper-Connections (HC) expand the residual stream into N parallel streams, giving a form of memory scaling.
Prior HC variants work well up to N=4, but performance gains flatten and training cost rises sharply beyond that.

Why scaling breaks

The authors identify two bottlenecks:

not enough write-back information as more streams are added
residual-mixing generation whose cost grows cubically with N

What xHC changes

Adds temporal feature augmentation to improve write-back
Uses a sparse residual-stream architecture that updates only k=4 out of N=16 streams, while still keeping dense access to the full residual state

Results

Across 18B and 28B MoE models, xHC gives consistent downstream gains.

On an 18B MoE model, it improves the average downstream score by 4.0 points over mHC
It also needs only modest extra training FLOPs relative to vanilla
Scaling-law experiments show vanilla and mHC need 1.50× and 1.19× the compute of xHC, respectively, to match the same loss

xHC-Flash

To make large-N training practical, the paper also introduces xHC-Flash, which cuts per-sublayer memory traffic from 73.5C to 40C, close to the 34C of mHC at N=4, while keeping the gains of full xHC.

The takeaway: xHC makes large-N residual-stream expansion both effective and more practical for LLM pre-training.

Original post →

xHC scales Transformer residual streams beyond N=4

More from Research