ByteDance Seed's Hierarchical Residuals decouple residual width from compute width in Transformers
ByteDance-Seed · hf · 2026-11-09
ByteDance Seed, with Tsinghua University, released a paper proposing Hierarchical Residuals (HiRes), a minimalist architectural change for Transformers.
Key points:
- Residual connections implicitly tie the residual width carried through depth to per-layer compute width.
- HiRes groups layers into blocks: standard d-dimensional PreNorm layers operate inside each block, while a wider D-dimensional PostNorm outer residual connects blocks via learned read/write projections.
- Across model sizes and depths, HiRes consistently outperforms vanilla residual addition, improving training efficiency and downstream accuracy, with fitted scaling laws indicating growing benefits.
More from Research
- Composer's experiment: AI agents fed unquantized music sweep top 3 in blind creativity judging — Music-For-Robots · 2026-11-09
- Boltzina v1.1 uses NVIDIA BioNeMo to speed molecular docking up to 24.4x vs Boltz-2 — AllThingsApx · 2026-11-09
- Chinese blog post breaks down conic-lifting paper and its implications for schedule-free learning — bremen79 · 2026-11-09
- StatQuest releases long-form tutorial: Linear Discriminant Analysis concepts and Python code — StatQuest with Josh Starmer · 2026-11-09
- KAIST paper identifies 'Teacher Drift' in multilingual reasoning distillation and teaches only when it helps — Jaewon Cho · 2026-11-09
- StatQuest releases long-form tutorial: PCA concepts and Python code — StatQuest with Josh Starmer · 2026-11-09