DIY dense-to-MoE upcycling on an 8GB GPU: Qwen2.5-0.5B MLP compute cut to ~40%

Bayliner1980 · reddit · 2026-10-11

A Reddit user shares experiments converting existing dense models into sparse Mixture-of-Experts models on an RTX 4060 (8GB), with no pretraining from scratch—a DIY version of the "upcycling" big labs do.

Method: converted Qwen2.5-0.5B and SmolLM2-360M into 32-expert top-8 MoEs with a shared expert, exported in Qwen2-MoE format; the hardest layers (final layers, SmolLM2's layer 3) were kept dense at a slight compute cost.

Results (accuracy drops across the board):

Qwen2.5-0.5B-MoE-A0.3B: 40% of original MLP compute per token; WikiText-2 ppl 14.66→19.20, mean accnorm 0.467→0.408
SmolLM2-360M-MoE-A0.2B: 44%; WikiText-2 ppl 12.94→19.11, mean accnorm 0.514→0.417

Limitations: using at most 50% of original MLP compute guarantees a quality gap, and at this tiny scale speedup gains are negligible. Next steps: larger models, longer training, finer expert layouts—all released openly.

Original post →

DIY dense-to-MoE upcycling on an 8GB GPU: Qwen2.5-0.5B MLP compute cut to ~40%

More from Research