Rednote's Dots unveils MSAR: 1.58x iso-loss compute speedup over residual baseline
teortaxesTex · x · 2026-10-14
A paper from Dots (Rednote) introduces MSAR (Multi-Stream Attention Residuals), generalizing Kimi's AttnRes to multi-head depth attention via cheap multi-scale causal depthwise convolutions. Block MSAR stores only single-stream block summaries, cutting storage from O(mNd) to O(Nd). On 18B/28B MoE pretraining it beats Vanilla, mHC, and AttnRes (avg 45.7 vs 37.5 at 18B), with the lowest final loss (1.750) and a 1.58x iso-loss compute speedup versus 1.35x for AttnRes and 1.30x for mHC.
Related event: SJTU and Xiaohongshu's Dots Propose MSAR Attention Architecture(3 posts)→
More from Infra
- Photonic Training of a 1B GPT-2 Without Backprop Lands in PNAS — IgorCarron · 2026-10-14
- Xinjiang's 167 GW green power, 16ms to Chongqing: the grid edge in AI race — 96Stats · 2026-10-14
- Migrating a key API from Node to Bun cut p99 latency 80% and memory 63% — DanielLockyer · 2026-10-14
- Agent traffic is a strange inference workload: long prefix caching beats raw tokens per second — rohanpaul_ai · 2026-10-14
- LightVAE speeds up video VAE decoding up to 2.24x on H100 with minimal quality loss — Shiqiao Gu · 2026-10-14
- Orbital data centers have no off-switch: Reddit debates AI safety paradox — Zappy_Oh · 2026-10-14