Fudan & Shanghai AI Lab map PEFT recipes for MoE LLMs: tune experts alone, LoRA/DoRA win
kalyan_kpl · x · 2026-10-20
A large-scale empirical study from Fudan University, Shanghai AI Laboratory and collaborators systematically establishes PEFT design recipes for MoE LLMs across three backbones. Key findings: adapting MoE experts alone gives the highest downstream performance with competitive general-capability retention at equal parameter budgets; native expert-weight updates beat adapters that bypass experts; LoRA and DoRA define the strongest operating region; no nonzero load-balancing coefficient Pareto-dominates the zero baseline. A weight-level redesign of MoE²-LoRA on Qwen3 moves the module-level bypass pool into native expert weights, improving four upstream endpoints by 2.4–17.9 points with 17% fewer trainable params at 3.2-point average downstream cost.
More from Models
- DeepSeek v4.1 flash model now live on Tinker — cHHillee · 2026-10-20
- The deslop experiment: 161 training answers on a MacBook Air zeroed out em dashes — wateriscoding · 2026-10-20
- DeepSeek and MiniMax Show Strikingly Similar Chain-of-Thought Styles, Sparking Distillation Talk — teortaxesTex · 2026-10-20
- Gemini Pro user rants: Argon usage cap hits after just 5 prompts — Atomicfarms · 2026-10-20
- MiniMax M3.1 Flash ranks #2 in human eval and trains another AI at far lower cost — my_cat_can_code · 2026-10-20
- Google ready to invest up to $40B in Anthropic, pending conditions — emmanuelvivier · 2026-10-20