SimpleHC matches mHC performance with only 2.4% extra training time, study finds
dots-studio · hf · 2026-11-13
A systematic study of Hyper-Connections (HC) architectures from SJTU, Xiaohongshu's Dots Studio, PKU and others identifies two effective interaction groups: learned read/write gating without mixing, and unconstrained mixing with fixed gates—each alone matches mHC performance. Repeated access to the expanded residual state is pinpointed as the main source of training overhead. The resulting SimpleHC removes residual mixing and simplifies execution via shared memory access: layer-specific gates share a group-entry state enabling batched reads and deferred writes, with scalar corrections exactly preserving sublayer contributions. At the largest tested scale (15B–30B MoE), SimpleHC adds only 2.4% end-to-end training time over a vanilla residual baseline, matches mHC performance, and shows smoother gradient dynamics.
More from Infra
- DeepSeek V4.1-Flash seeds resist 4.3B-seed search; Huawei chip shift rumored — TheNormanMu · 2026-11-13
- OpenCartridge's trust bill: no auth, public bucket, Docker socket exposed — maier_ak · 2026-11-13
- camenduru open-sources OpenCartridge, packaging AI models as one-click 'cartridges' — maier_ak · 2026-11-13
- Game assets on your own GPU: what OpenCartridge does and what it asks you to trust — maier_ak · 2026-11-13
- Core Memory podcast: Sceye's $100M high-altitude airship wins SoftBank fleet order for Japan broadband — kyliebytes · 2026-11-13
- Torus explains closed-loop liquid cooling: no on-site water hookup needed — ccerrato147 · 2026-11-13