MSAR Multi-Stream Attention Residuals Outperform Existing Methods on MoE Models

SJTU, Xiaohongshu Dots Studio and other institutions propose MSAR, a multi-stream attention residuals architecture that comprehensively outperforms AttnRes and mHC on 18B/28B MoE models, achieving 1.58x speedup at equal-loss compute.

2026-10-13 ~ 2026-10-14 · 2 related posts

• MSAR multi-stream attention residuals beat prior variants at 18B/28B MoE, 1.58x iso-loss speedup — SJTU · 2026-10-13
• MSAR brings multi-head depth attention to residuals, beating AttnRes and mHC with 1.58x iso-loss speedup — SonglinYang4 · 2026-10-14