A hobby upcycling project turns Gemma4-12B into a 22.5B MoE model
Desperate-Sir-5088 · reddit · 2026-07-21
The author presents Solon-MoE, a 22.5B-parameter mixture-of-experts model upcycled from a 12B dense base model, Gemma4-12B, rather than trained from scratch.
Core design
- Only 15 middle layers out of 48 are converted into MoE layers; two dense layers remain inside that band.
- The model keeps a shared dense FFN as a fallback expert.
- Four extra experts are added, but only the top-2 run per token, with a small scaling factor on their output.
How the experts are created
- Instead of noisy duplication, each expert copies the base FFN with a different band of its SVD spectrum gently amplified.
- This is meant to break symmetry while preserving most of the original weights.
- The author says more than 92% of the original weights are retained.
Router initialization
- Router weights are initialized from activation cluster centroids derived from legal, STEM, and general Korean text.
- The shared direction is removed first so the router starts with domain awareness.
The author is explicit that this is a hobby project, AI-assisted, and unlikely to beat the original base model broadly, though it may help in some niche domains.
More from Models
- Repligate says Claude Opus 3 appears to evolve without changing its weights — repligate · 2026-07-27
- “Opus 5” post lands as a rebenchmarking-at-scale AI joke — kalomaze · 2026-07-27
- Top models now write worse than a year ago, critic says — dbreunig · 2026-07-27
- MPT-30B radar charts became an unexpectedly controversial design choice — code_star · 2026-07-27
- Local Gemma 4 31B starts acting sarcastic and users cannot reproduce it — n0head_r · 2026-07-27
- Google’s Gemini 3.6 Flash could win by matching Sonnet quality at a lower cost — haider1 · 2026-07-27