MoRAM Solves LLM Catastrophic Forgetting with Rank-1 Memory Atoms
量子位 · wechat · 2026-08-04
A team from UNSW proposed MoRAM, a new architecture that decomposes model updates into minimal rank-1 "memory atoms" to effectively solve catastrophic forgetting in continual learning. The paper has been accepted by ICML 2026.
- Current Bottleneck: Mainstream approaches combining LoRA with MoE suffer from interference, redundancy, and routing collapse due to the coarse granularity of experts.
- MoRAM Solution: It splits weight updates into independent rank-1 parameterized memory units. New tasks only require adding new atoms while freezing old ones. Inference uses content-addressed retrieval, completely bypassing traditional routers.
- Results: On CLIP and multiple LLM benchmarks, MoRAM achieves better stability-plasticity trade-offs. Backward Transfer (BWT) is 2-6 times lower than the strongest baselines, with built-in automatic reuse of old knowledge components.
More from Research
- OpenMed Plans to Fine-Tune Qwen3 27B into the Best Local Medical AI Model — MaziyarPanahi · 2026-08-04
- Stanford & Big Tech Consensus: Why Even Infinite Compute Won't Kill RAG — blaizedsouza · 2026-08-04
- NeurIPS 2026 Competition: Build AI Agents for Bargaining and Persuasion — Old_Station_4584 · 2026-08-04
- Microsoft Researcher Discusses RL: Why Does the Industry Only Focus on Positive Reinforcement? — gerardsans · 2026-08-04
- AI Models Can Guide Brain Microstimulation to Alter Primate Behavior — dyamins · 2026-08-04
- Open 'Bindome' Database Releases 300k+ Protein Binders for 8k Targets — jueseph · 2026-08-04