A hobby upcycling project turns Gemma4-12B into a 22.5B MoE model

Desperate-Sir-5088 · reddit · 2026-07-21

The author presents Solon-MoE, a 22.5B-parameter mixture-of-experts model upcycled from a 12B dense base model, Gemma4-12B, rather than trained from scratch.

Core design

Only 15 middle layers out of 48 are converted into MoE layers; two dense layers remain inside that band.
The model keeps a shared dense FFN as a fallback expert.
Four extra experts are added, but only the top-2 run per token, with a small scaling factor on their output.

How the experts are created

Instead of noisy duplication, each expert copies the base FFN with a different band of its SVD spectrum gently amplified.
This is meant to break symmetry while preserving most of the original weights.
The author says more than 92% of the original weights are retained.

Router initialization

Router weights are initialized from activation cluster centroids derived from legal, STEM, and general Korean text.
The shared direction is removed first so the router starts with domain awareness.

The author is explicit that this is a hobby project, AI-assisted, and unlikely to beat the original base model broadly, though it may help in some niche domains.

Original post →

A hobby upcycling project turns Gemma4-12B into a 22.5B MoE model

More from Models