ICLR 2026 paper MoM: multiple memory states fix linear models' recall weakness
kastnerkyle · x · 2026-09-12
An ICLR 2026 poster introduces Mixture-of-Memories (MoM), targeting a key limitation of linear sequence models (linear attention, SSMs, linear RNNs): compressing the whole sequence into a single fixed-size memory state hurts recall-intensive tasks.
- MoM keeps multiple independent memory states, with a router directing tokens to specific states, boosting memory capacity while reducing interference
- Works as a general framework that plugs into diverse memory update mechanisms across linear models
- Each memory state stays linear-complexity: linear training cost, constant inference cost
- Experiments show MoM beats existing linear sequence models on downstream tasks, especially recall-intensive ones, approaching Transformer-level performance
More from Research
- Aligning image and text embeddings without paired data: Wasserstein Procrustes and rotation papers — phillip_isola · 2026-10-31
- Phillip Isola demos aligning image and text embeddings with just a single rotation — phillip_isola · 2026-10-31
- Arena Offers Up to $50,000 Per Project for Academic Research on AI Evaluation — arena · 2026-10-31
- StudentBench: AI tutoring matches human GRE gains at 918x lower cost — gharik · 2026-10-31
- WISD: Distill Training Signal from Weaker Quantized Copies of Reasoning Models — jet-ai · 2026-10-31
- Albert Gu: H-Net learns to segment Chinese characters from raw bytes mid-training — niloofar_mire · 2026-10-31