SJTU's MSAR Residual Architecture Delivers 1.58x Iso-Loss Training Speedup Over Standard Residuals
burny_tech · x · 2026-10-20
Researchers from Shanghai Jiao Tong University and collaborators introduce Multi-Stream Attention Residuals (MSAR), addressing information dilution in standard residual streams as models deepen.
How it works
- Standard residual connections squeeze all prior layers into one hidden state; MSAR generalizes Attention Residuals (AttnRes) into multi-head attention along the depth axis.
- Multi-scale causal convolutions give heads distinct contextual views for retrieving historical representations, combined via lightweight adaptive mixing into a single-stream layer input — no persistent multi-stream storage or communication needed.
Results
- Outperforms standard residuals, mHC, and AttnRes across all 12 benchmarks at 18B and 28B MoE scales.
- Scaling-law experiments show a 1.58x iso-loss compute speedup over standard residuals (vs 1.35x for AttnRes and 1.30x for mHC), with only 0.5% extra training FLOPs per step.
More from Research
- Directly supervising representation dot products with generative-model PMI estimates hailed as a new paradigm — vincesitzmann · 2026-10-20
- Logarithmic returns come from opportunities spread across orders of magnitude of cost — DimitrisPapail · 2026-10-20
- Emergent curriculum without labels: prompt weights diverge 1.6x vs 0.75x over training — shangbinfeng · 2026-10-20
- Stackelberg Alignment turns LLM disagreement into a training curriculum via bandit leader — shangbinfeng · 2026-10-20
- RL fine-tuning Qwen3.5-0.8B for under $20 on HF Jobs lifts pass@1 from 10% to 36% — LysandreJik · 2026-10-20
- Two New Robotics Benchmarks Released: MicroPlanBench and MicroAnnotationBench First Rankings — chris_j_paxton · 2026-10-20