MSAR multi-stream attention residuals beat prior variants at 18B/28B MoE, 1.58x iso-loss speedup
SJTU · hf · 2026-10-13
Researchers from SJTU, Xiaohongshu's Dots Studio, and PKU introduce Multi-Stream Attention Residuals (MSAR), generalizing Attention Residuals to multi-head depth attention. Multi-scale causal convolutions supply stream diversity; retrieved streams are combined via lightweight adaptive mixing without persistent multi-stream storage. Across all 12 benchmarks at 18B/28B MoE scale, MSAR outperforms standard residuals, mHC, and AttnRes, with a 1.58x iso-loss compute speedup over the standard residual baseline (vs 1.35x AttnRes, 1.30x mHC).
More from Research
- Reinventing the wheel: hand-rolled upscaling for tiny retro-console pixel sprites — ssh4net · 2026-10-13
- Preferred Networks' T3lescope reconstructs high-fidelity 3D scenes from multi-view images without per-scene optimization — pfnet · 2026-10-13
- ARCS benchmark: ambiguous text-to-SQL stumps models, best at just 51% accuracy — megagonlabs · 2026-10-13
- Alibaba's TerminalHorizon synthesizes long-horizon agent data, lifting Qwen3.5-27B across 12 benchmarks — alibabagroup · 2026-10-13
- Meta's Loop Scaling Laws: 5-pass 0.3B MoE matches 2x larger dense-active MoE — ChengleiSi · 2026-10-13
- LLM discovery loops suffer mode collapse; warm-starting from strong initial populations fixes it — BenBlaiszik · 2026-10-13