MHAR Improves Transformer Residuals, Cutting Validation Loss at 1B Scale
Cheng Luo · hf · 2026-07-31
Standard Transformers propagate information via a single residual stream, forcing feature subspaces to compromise on reading depth history as model width grows. This paper introduces Multi-Head Attention Residuals (MHAR), which reshapes the routing query into multiple per-subspace heads, each with its own softmax over depth history.
The method adds zero parameters and negligible compute. Trained from scratch at 100M, 350M, and 1B scales, MHAR consistently outperforms standard Transformers and other baselines, with gains scaling alongside model size. The head count acts as a true design axis, showing a U-shaped validation loss curve with an optimum at H=4 or H=8. Fused Triton kernels improve training throughput, and an identity-preserving conversion applied during 8B mid-training yields improvements of +3.2 on GSM8K and +3.1 on GPQA.
Related event: MHAR Improves Transformer Residuals to Reduce Validation Loss(2 posts)→
More from Research
- Qdrant tests Perplexity's visual PDF embeddings: 21x smaller vectors, 99.9% after rescoring — jrxna · 2026-10-16
- Tessellation-free geometric queries on displaced surfaces cut caustics variance 10x — ssh4net · 2026-10-16
- MIT's Buehler: AI as 'a new language of science,' three years on — ProfBuehlerMIT · 2026-10-16
- Mere availability of AI advice collapses human "I don't know" from 43% to 3% — ValerioCapraro · 2026-10-16
- One Bias Score, Two Factors: POSTECH Paper Decomposes Option Bias in MC Evaluation — Hyunjong Ok · 2026-10-16
- When the KV Cache Is Slower Than the GPU: Remote KV Tiers Backfire in Production — MultiverseComputingCAI · 2026-10-16