MSAR residual architecture delivers 1.58x iso-loss speedup at 18B/28B scale
LChoshen · x · 2026-11-08
A Weizmann Institute reading group (#ActiveAIReading) reviewed the paper "Multi-Stream Attention Residuals" (Qin et al., 2026).
- MSAR extends Attention Residuals into multi-head depth attention, drawing stream diversity via multi-scale causal convolutions and combining retrieved streams with lightweight adaptive mixing
- It requires no persistent multi-stream storage or communication, with minimal overhead
- Across all 12 downstream benchmarks at 18B and 28B MoE scales, MSAR beats standard residuals, mHC (Hyper-Connections), and AttnRes
- Scaling-law experiments show a 1.58x iso-loss compute speedup over the standard residual baseline, versus 1.35x for AttnRes and 1.30x for mHC
Related event: MSAR Multi-Stream Attention Residuals Speed Up Compute by 1.58x(2 posts)→
More from Research
- IEEE retracts 1,417 conference papers in one day, the largest bulk retraction on record — asusarla · 2026-11-08
- Agents can reason about why they won't comply, so black-box behaviorism fails — teortaxesTex · 2026-11-08
- LLM named as co-author on weight-space programming study matching gradient optimization in 30/87 settings — lateinteraction · 2026-11-08
- Benchmark contamination checks aren't new: researchers debate cheap robustness tests — teortaxesTex · 2026-11-08
- RL staleness in agentic training: >100-step off-policy regimes favor labs with huge compute — teortaxesTex · 2026-11-08
- RoboCoach uses a world model to target weak subskills, lifting robot success 13.3% to 75% — jiqizhixin · 2026-11-08