Google's Sleep Framework Lets LLMs Keep Learning Without Catastrophic Forgetting
dl_weekly · x · 2026-11-05
Google Research published a blog on a "sleep framework" for continual learning: LLMs alternate between active interaction (wake) and offline memory consolidation (sleep), distilling fragile short-term context into new low-rank MoE capacity as stable long-term parameters.
- Motivation: deployed LLMs have fixed knowledge cutoffs; continual parameter updates cause catastrophic forgetting, while ICL only persists within the context window
- Inspired by how the human brain stabilizes memory during rest
- Results: retains long-context knowledge up to 10M tokens, continuously acquires new facts and improves complex reasoning
- Scores: 48.9% on SQuAD, 79.2% on AIME 24
A new paradigm exploring how to mitigate knowledge cutoffs and catastrophic forgetting.
More from Research
- Visual Token Routing cuts VLM agent visual tokens by up to 5.8x at same accuracy — wellecks · 2026-11-05
- Flex-RSI: a new benchmark measuring recursive self-improvement of frontier agents — _vztu · 2026-11-05
- Four generations of protein design models chart the road to fully programmable protein design — AllThingsApx · 2026-11-05
- ProteinMPNN likely memorized transmembrane folds; 0.3 noise breaks the recognition — sokrypton · 2026-11-05
- lm-eval Pitfall: Chatty Models Break Letter-Only Prompts, Regex Grabs Wrong Answers — mancubus77 · 2026-11-05
- Evolution can't copy homework: why species improve slower than iPhones — louisvarge · 2026-11-05