MHAR Improves Transformer Residuals, Cutting Validation Loss at 1B Scale
Cheng Luo · hf · 2026-07-31
Standard Transformers propagate information via a single residual stream, forcing feature subspaces to compromise on reading depth history as model width grows. This paper introduces Multi-Head Attention Residuals (MHAR), which reshapes the routing query into multiple per-subspace heads, each with its own softmax over depth history.
The method adds zero parameters and negligible compute. Trained from scratch at 100M, 350M, and 1B scales, MHAR consistently outperforms standard Transformers and other baselines, with gains scaling alongside model size. The head count acts as a true design axis, showing a U-shaped validation loss curve with an optimum at H=4 or H=8. Fused Triton kernels improve training throughput, and an identity-preserving conversion applied during 8B mid-training yields improvements of +3.2 on GSM8K and +3.1 on GPQA.
More from Research
- Review Paper on Reasoning Shortcuts in Neuro-Symbolic AI Published in JAIR — tetraduzione · 2026-07-31
- OpenMLE: Open-Source AI4AI System Hits 71% on MLE-Bench Lite Using a Single RTX 4090 — TianbaoX · 2026-07-31
- ShadowDancer: Teaching Video World Models Any Action via Shadow Pairs — AlayaLab · 2026-07-31
- Study Shows Late Interaction Models Generalize Better in Multilingual Tasks — lateinteraction · 2026-07-31
- Developer Open-Sources 'Unbiased' LLM Benchmark — cephaloform · 2026-07-31
- 3D Chest CT Vision-Language Models for Automated Report Generation Accepted at RSNA 2026 — maier_ak · 2026-07-31