CP-MoE: Freeze the Base, Tune Just 1.5% Params, No Forgetting
flosalim · x · 2026-08-21
Introducing CP-MoE (Consistency-Preserving Mixture-of-Experts): a post-training framework that fine-tunes LLM/VLM models by updating less than 1.5% of parameters while keeping the 7B base model fully frozen, all while avoiding catastrophic forgetting—traditional efficient fine-tuning methods (such as LoRA) continually erase prior task knowledge when tasks are extended sequentially, due to parameter overwriting and rigid routing load balancing.
The core strategy is "assess before updating": a temporary expert—a disposable expert—first adapts during a brief warm-up phase on the new task, and its learning trajectory then guides the entire learning process before being discarded, adding no long-term overhead. This is how CP-MoE controls the routing and update scheme of each LoRA expert.
More from Models
- Speech-to-Speech Model Analysis: Grok Leads Task Success, Gemini Top Preference — ArtificialAnlys · 2026-08-21
- Speech Agent Arena Launch: Gemini Leads Preference, Grok Leads Success Rate — ArtificialAnlys · 2026-08-21
- NVIDIA's Coding Agent Scores 100% on ARC-AGI-3 Benchmark — MagicZhang · 2026-08-21
- Tom Yeh Uploads Kimi 3 Seminar Recording, with Nathan Lambert on His Moonshot Visit — ProfTomYeh · 2026-08-21
- Gemini Flash outputs unrecognizable text, sparking 'alien language' theories — MSCharan_ · 2026-08-21
- NVIDIA AVO achieves 100% on ARC-AGI benchmark — theologi · 2026-08-21