Hyper-Connection Factory: open-source repo benchmarks 11 residual-connection variants on one LLM
ChengleiSi · x · 2026-10-04
hyper-connection-factory is an open-source repo that implements 11 hyper-/residual-connection variants for decoder LLMs — HC, mHC, Frac, DenseFormer, MUDDFormer, LAuReL, AttnRes, MHAR, DAR and more — behind a single interface.
- Every variant shares the exact same attention/MLP sublayers, initialization, data, optimizer and schedule; only the Connection module (what each sublayer reads and how outputs are written back) changes, so result differences are attributable purely to the wiring.
- All variants are trained under one recipe from 38M to 1.2B params at 20 tokens/param, with configs, scripts, results and a per-method table covering paper references and extra-parameter overhead (e.g. postnorm adds 2L·d params; HC uses parallel residual streams).
More from Research
- Broad, Boston Children's and Jackson Lab launch Center for Therapeutic Genetics for rare disease medicines — anshulkundaje · 2026-10-04
- Market Basket Analysis tells you what sells, not who is leaving — dima806_dima · 2026-10-04
- Heavy AI Users Score Worse on Critical Thinking Tests: Researchers Call It Cognitive Surrender — DrKavner · 2026-10-04
- NBER paper argues deep learning is a projection method, merging them yields better solvers — SchoeneggerPhil · 2026-10-04
- Four years of AI progress across seven capabilities, adjusted for benchmark changes — epheva · 2026-10-04
- Arthur Gretton to Talk on Gradient Flows on MMD at NYC Probabilistic Modeling Workshop — ArthurGretton · 2026-10-04