ICLR 2026 paper MoM: multiple memory states fix linear models' recall weakness
kastnerkyle · x · 2026-09-12
An ICLR 2026 poster introduces Mixture-of-Memories (MoM), targeting a key limitation of linear sequence models (linear attention, SSMs, linear RNNs): compressing the whole sequence into a single fixed-size memory state hurts recall-intensive tasks.
- MoM keeps multiple independent memory states, with a router directing tokens to specific states, boosting memory capacity while reducing interference
- Works as a general framework that plugs into diverse memory update mechanisms across linear models
- Each memory state stays linear-complexity: linear training cost, constant inference cost
- Experiments show MoM beats existing linear sequence models on downstream tasks, especially recall-intensive ones, approaching Transformer-level performance
More from Research
- Open-weights SUPlime beats pyannote's commercial diarization with 15.86 DER — solyarisoftware · 2026-09-12
- AI out-persuades world champion debaters, raises donations nearly 3x better than pros — ben_j_todd · 2026-09-12
- Shifting local token interactions from inference to training-time lookups seen as a scaling win — AccBalanced · 2026-09-12
- Open-source RL for large MoEs with zero train-infer mismatch, teaching Qwen3.6-35B-A3B to play Wordle — kastnerkyle · 2026-09-12
- Harvard researcher turned a fruit fly brain connectome into a Bitcoin trading bot — Scobleizer · 2026-09-12
- LLMs ace hard problems but miss the basics: KST study of 8 models vs 18,000 humans — rohanpaul_ai · 2026-09-12