ToMoE paper converts dense LLMs to MoE without fine-tuning

pmttyji · reddit · 2026-08-24

ToMoE proposes a method to convert dense Large Language Models into Mixture-of-Experts (MoE) models through dynamic structural pruning. This approach reduces the number of active parameters without permanently deleting them, converting MLP layers into a MoE architecture to maintain a fixed count of active parameters. Experiments show it consistently outperforms previous structural pruning techniques on model families like Phi-2, LLaMA-2/3, and Qwen-2.5, even without fine-tuning. The code is open-sourced on GitHub.

Original post →

ToMoE paper converts dense LLMs to MoE without fine-tuning

More from Models