SimpleHC cuts Hyper-Connections training overhead from 59–100% to just 2.4%
SonglinYang4 · x · 2026-11-09
A new arXiv paper systematically dissects Hyper-Connections (HC) architectures, showing that existing implementations like DeepSeek's mHC add 59–100% training overhead in PyTorch — largely from repeated access to the expanded residual state.
The authors identify two effective interaction groups: learned read/write gating without mixing, or unconstrained mixing with fixed gates — either alone matches mHC performance. This leads to SimpleHC, which drops residual mixing and simplifies execution via shared memory access: layer-specific gates share a group-entry state enabling batched reads and deferred writes.
Results: only 2.4% end-to-end training overhead over vanilla residual at the largest tested width, with performance comparable to mHC when scaling from 15B to 30B MoE models. Songlin Yang notes the scheme amounts to chunkwise linear attention — ((q^T k)v) computed in parallel per block without materializing the expanded hidden state.
More from Models
- The 'Jensen loves Chinese models' joke: not the LLMs everyone assumed — haider1 · 2026-11-09
- Weight-Space Programming replicates: hand-edited 18GB 'Nanseek' nears frontier scores at 1/25th the size — teortaxesTex · 2026-11-09
- Meta retrains its everyday models on flagship Muse Wonder: 2.4x less forgetting, +67% task completion — xiaohu · 2026-11-09
- Daily AI brief: stealth 1M-context model Wyvern spotted in Devin, Anthropic locks $3.4B power deal — testingcatalog · 2026-11-09
- Qwen trained to continuously refit happiness probes — Sauers_ · 2026-11-09
- Qwen teases Qwen4 with 'Think Begins' poster; Reddit begs for the tensors — Ok-Shower7286 · 2026-11-09