Multi-Head Attention Residuals Enhance Transformer Depth Routing
heghbalz · x · 2026-08-02
Traditional Transformers propagate information across depth using a single additive residual stream, forcing all feature subspaces to share one routing query. This compromise degrades performance as model width grows.
The paper Multi-Head Attention Residuals (MHAR) introduces a new architecture to fix this:
- Mechanism: Reshapes the routing query into H per-subspace heads, each with its own softmax over depth history.
- Zero Parameters: It is essentially a block-diagonal reshape, adding zero parameters and negligible compute.
- Performance: Outperforms standard Transformers at 100M, 350M, and 1B scales, with validation loss improvements increasing with model size.
- Optimal Heads: The head count H is a real design axis. H=4 or H=8 yields optimal results, while over-splitting (H=16) degrades performance.
Related event: MHAR Improves Transformer Residuals to Reduce Validation Loss(2 posts)→
More from Research
- DeepMind launches Antigravity for Science: coding agent built for end-to-end research workflows — pushmeet · 2026-10-16
- It's not gradient descent: burny_tech corrects the diagnosis to RLHF mode collapse — burny_tech · 2026-10-16
- A beginner-friendly introduction to causal inference kicks off, Part 1 — Nasereliver · 2026-10-16
- FBI method trains for Pass@K with one sample per update at 12.3% of the cost — EmtiyazKhan · 2026-10-16
- Berkeley researchers sample humanoid robot policies directly from weight space with diffusion — berkeley_ai · 2026-10-16
- OneWM: unified causal training shows world and action modeling improve each other — IAAI233 · 2026-10-16