Google and Cornell Use Sleep-Inspired MoE Distillation to Fix Catastrophic Forgetting
jmugan · x · 2026-11-03
A Google Research and Cornell paper, 'Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories,' borrows the hippocampus-to-cortex memory consolidation mechanism to fight catastrophic forgetting with mixture-of-experts:
- Accumulate memories as text in the context window.
- During 'sleep,' use distillation to write knowledge from the full network plus context into a temporary expert.
- During 'dreaming,' randomly activate long-term experts to train and blend in the knowledge, freeing the temporary expert for the next cycle.
More from Research
- kalomaze conjectures deep sequence models could mimic restricted training families like rank-1 attention LoRAs — kalomaze · 2026-11-03
- Deep Principle releases open-weight science model Hydrogen-1.0, beating GPT-6-Astra on FrontierScience — dair_ai · 2026-11-03
- Zyphra's ZHA brings subquadratic attention via hierarchical tree-structured indexer — yacineMTB · 2026-11-03
- kalomaze: In-context learning is an incidental byproduct of next-token prediction, not deliberate design — kalomaze · 2026-11-03
- RSNA 2026 lab walks through fine-tuning a VLM for radiology end to end — PRouzrokh · 2026-11-03
- RSNA 2026 lab walks through fine-tuning a VLM for radiology end to end — PRouzrokh · 2026-11-03