Google Research's sleep framework lets LLMs consolidate up to 10M tokens into long-term memory
rseroter · x · 2026-10-29
Google Research (Ali Behrouz, Vahab Mirrokni) introduces the "sleep framework" for continual learning, tackling catastrophic forgetting from continual parameter updates and the session-bound nature of in-context learning.
Key ideas:
- Wake/sleep alternation: an active phase captures new info as fast short-term memory (KV cache); an offline sleep phase consolidates fragile short-term tokens into stable long-term parametric memory.
- Consolidation: uses low-rank adaptation, mix-of-experts, and specialist model processing.
- Dreaming: synthetic routing generates random experts, refined with reinforcement learning.
Google reports the framework lets models continuously acquire new facts, retain long-context knowledge up to 10M tokens, and improve complex reasoning.
Related event: Google Research Proposes "Sleep Framework" for Continual Learning in LLMs(8 posts)→
More from Research
- New paper challenges JEPA feature-selection theory: unpredictable features can be kept — andreea7b · 2026-10-29
- SREGym to debut at Bench Summit, targeting AI SRE evaluation frontier — tianyin_xu · 2026-10-29
- AA-Robotics benchmark: GPT-6 Astra tops robot-arm control, still ~100x slower than humans — ArtificialAnlys · 2026-10-29
- Delta-Matching closes the FP8 attention gap: all 7 core matmuls in FP8, loss matches BF16 — songhan_mit · 2026-10-29
- RFD4-Proteina lands on Hugging Face as HF publishes its October AI for Science roundup — AllThingsApx · 2026-10-29
- Google's PLE: 85 real engineering tasks, best agent config scores only 50.6% — rohanpaul_ai · 2026-10-29