ToMoE paper converts dense LLMs to MoE without fine-tuning

pmttyji · reddit · 2026-08-24

ToMoE proposes a method to convert dense Large Language Models into Mixture-of-Experts (MoE) models through dynamic structural pruning. This approach reduces the number of active parameters without permanently deleting them, converting MLP layers into a MoE architecture to maintain a fixed count of active parameters. Experiments show it consistently outperforms previous structural pruning techniques on model families like Phi-2, LLaMA-2/3, and Qwen-2.5, even without fine-tuning. The code is open-sourced on GitHub.

Original post →

More from Models

• Meta open-sources Muse Spark Open: 816B-parameter model that runs locally on two Mac Studios — gabriberton · 2026-10-16
• Grok Voice 3.0 Tops Design Arena Speech Leaderboard, Up 7.4 Points — XFreeze · 2026-10-16
• Robotics LLMs tested on 16 tasks: GPT-6 Astra hits 17.2% success vs Fable's 9.1% — pabbeel · 2026-10-16
• Meta open-sources Muse Spark: top US open model, scores just below DeepSeek — bindureddy · 2026-10-16
• TRL adds day-0 support for fine-tuning Muse Spark Open's 816B MoE with LoRA on 64 H100s — QGallouedec · 2026-10-16
• Meta's Muse Spark Open ships detailed eval docs; researcher pushes machine-readable EEE standard — evijit · 2026-10-16

Models channel →