Sakana AI's Continuous Memory Machine splits short-term compute from long-term storage

dair_ai · x · 2026-10-09

Sakana AI published a paper on memory for recurrent models, the Continuous Memory Machine (CMM).

• Problem: Recurrent models track state well, but short-term computation and long-term storage compete for the same hidden vector.
• Method: CMM uses two memory matrices — one tracking recent neuron activity, one storing information for later steps — with a Transformer reading and writing both at every step. It builds on the Continuous Thought Machine.
• Results: Beats LSTM, DNC, RMC, and CTM on copy, associative recall, sorting, few-shot regression, and maze solving, and generalizes to longer inputs than prior memory-augmented networks.
• Interpretability: Attention maps show long-term memory is engaged for algorithmic and in-context tasks and skipped otherwise.

Original post →

More from Research

• Could Looped Models Resist Distillation Attacks by Reasoning in Latent Space? — moyix · 2026-10-09
• LLM-as-a-Verifier: Weaker Model Verifies Stronger One, Hits 69.2% SOTA on Terminal-Bench 4 — Azaliamirh · 2026-10-09
• Salesforce's SRD distills hindsight into foresight, lifting 2B agent success from 0% to 60.6% — Salesforce · 2026-10-09
• Claude claims discovery of new binary red dwarf pair ~500 light-years away — nitarshan · 2026-10-09
• Prompt Tuning Is Forgotten Lore — Are We Massively Underusing Finetuned Tokens? — cephaloform · 2026-10-09
• Ricardo Baeza-Yates Lecture: When Will ML Evaluation Stop Fooling Itself? — PolarBearby · 2026-10-09

Research channel →