MSAR brings multi-head depth attention to residuals, beating AttnRes and mHC with 1.58x iso-loss speedup
SonglinYang4 · x · 2026-10-14
Researchers from Shanghai Jiao Tong University, Xiaohongshu's Dots Studio and Peking University introduce Multi-Stream Attention Residuals (MSAR), extending AttnRes from single-stream to multi-head attention along the depth axis. Each head retrieves earlier representations through a distinct multi-scale causal convolution context view, combined via lightweight adaptive mixing — no persistent multi-stream storage or communication required.
Key results:
- Consistently beats standard residuals, mHC and AttnRes across all 12 benchmarks at 18B and 28B MoE scales, adding only 0.5% training FLOPs at 28B
- Scaling-law experiments yield a 1.58x iso-loss compute speedup over the standard residual baseline, vs 1.35x for AttnRes and 1.30x for mHC
The authors position MSAR as a promising residual architecture for large-scale pre-training; the project is open-sourced on GitHub.
More from Infra
- Eaton's 800 VDC power rack draws big crowds at OCP summit — firstadopter · 2026-10-14
- Mixed-precision, mixed-accelerator samplers cut RL training cost with off-policy correction — xiuyu_l · 2026-10-14
- Pairing M5 Ultra 256GB with M3 Ultra 512GB for disaggregated prefill/decode cuts TTFT from 40s to 18s — Usual_Tackle5892 · 2026-10-14
- Vercel cuts build prep time 20%, and 54% for Git LFS projects — cramforce · 2026-10-14
- IBM, RIKEN and Cleveland Clinic simulate 12,635-atom protein with quantum-classical hybrid — aronchick · 2026-10-14
- Ninfer-ext: from-scratch C++/CUDA engine hits 2,142 tok/s decode on Qwen 35B nvfp4 with one RTX 5090 — giveen · 2026-10-14