A hobby upcycling project turns Gemma4-12B into a 22.5B MoE model
Desperate-Sir-5088 · reddit · 2026-07-21
The author presents Solon-MoE, a 22.5B-parameter mixture-of-experts model upcycled from a 12B dense base model, Gemma4-12B, rather than trained from scratch.
Core design
- Only 15 middle layers out of 48 are converted into MoE layers; two dense layers remain inside that band.
- The model keeps a shared dense FFN as a fallback expert.
- Four extra experts are added, but only the top-2 run per token, with a small scaling factor on their output.
How the experts are created
- Instead of noisy duplication, each expert copies the base FFN with a different band of its SVD spectrum gently amplified.
- This is meant to break symmetry while preserving most of the original weights.
- The author says more than 92% of the original weights are retained.
Router initialization
- Router weights are initialized from activation cluster centroids derived from legal, STEM, and general Korean text.
- The shared direction is removed first so the router starts with domain awareness.
The author is explicit that this is a hobby project, AI-assisted, and unlikely to beat the original base model broadly, though it may help in some niche domains.
More from Models
- Fangcun AI, China's first independent frontier AI safety lab, launches with DeepSeek eval report — JacquesThibs · 2026-10-26
- Kimi K3.1 Pro tops MathArena open models, closing frontier gap to about 6 months — teortaxesTex · 2026-10-26
- Ruff creator Charlie Marsh: frontier models find infinite bugs in any software — charliermarsh · 2026-10-26
- Dev testing one skill across Opus 5.5 and Sol 6.1 finds instruction following 'all over the map' — lucasmeijer · 2026-10-26
- supermemory unveils learner-1, claiming continual learning for any model — julianweisser · 2026-10-26
- Anthropic says Opus 5.5 cuts cost per task by ~40% vs Opus 5 — taufiqintech · 2026-10-26