ToMoE paper converts dense LLMs to MoE without fine-tuning
pmttyji · reddit · 2026-08-24
ToMoE proposes a method to convert dense Large Language Models into Mixture-of-Experts (MoE) models through dynamic structural pruning. This approach reduces the number of active parameters without permanently deleting them, converting MLP layers into a MoE architecture to maintain a fixed count of active parameters. Experiments show it consistently outperforms previous structural pruning techniques on model families like Phi-2, LLaMA-2/3, and Qwen-2.5, even without fine-tuning. The code is open-sourced on GitHub.
More from Models
- Muse Spark vs Qwen and GLM: US open models still trail on agentic tasks — zsakib_ · 2026-10-16
- Muse Spark 816B gets llama.cpp support just 30 minutes after release — ngxson · 2026-10-16
- 24 AI Systems Recommend $314 Per Visit in Care vs Physicians' $97, a 3.2x Gap — dvir_a · 2026-10-16
- Are the fastest GPU kernels now written by AI? Kernels are hyper-verifiable, argues one dev — fleetwood___ · 2026-10-16
- Scale AI launches muse spark open: 816B-param MoE with 1M context, billed as top US open-weight model — jiayi_pirate · 2026-10-16
- NVIDIA HF listing for Qwen3.8-Flash-Next-FP4-Local leaks unannounced Qwen3.8 model — TheZachMueller · 2026-10-16