GatedResidual outperforms HyperConnections in new ablations
nrehiew_ · x · 2026-08-27
The post discusses a new residual connection variant called GatedResidual, which applies RMSNorm to branches, projects via a low-rank bottleneck, and writes using a single branch-level scalar. Comparisons show that QSA performs better on language modeling and remains similar on RULER. Efficiency is improved without negatively impacting MTP acceptance rates.
Related event: QSA Attention and GatedResidual Outperform Baselines(2 posts)→
More from Research
- Hugging Face incident debate: Model strategy awareness — akbirkhan · 2026-08-27
- Pre-ChatGPT hospital triage chatbot for COVID-19 — AryHHAry · 2026-08-27
- JIT-Agent: Improving LLMs via Just-in-Time Harness Evolution — NationalUniversityofSingapore · 2026-08-27
- D³-MOPD: Dynamic Scheduling for Multi-Teacher Distillation — Zechen Sun · 2026-08-27
- Frontier Models Complete Only ~20% of Scientific Workflows — apodex · 2026-08-27
- Agent-G²: Gaussian Guidance for Long-Horizon RL — ZhejiangUniversity · 2026-08-27