A hobby upcycling project turns Gemma4-12B into a 22.5B MoE model
Desperate-Sir-5088 · reddit · 2026-07-21
The author presents Solon-MoE, a 22.5B-parameter mixture-of-experts model upcycled from a 12B dense base model, Gemma4-12B, rather than trained from scratch.
Core design
- Only 15 middle layers out of 48 are converted into MoE layers; two dense layers remain inside that band.
- The model keeps a shared dense FFN as a fallback expert.
- Four extra experts are added, but only the top-2 run per token, with a small scaling factor on their output.
How the experts are created
- Instead of noisy duplication, each expert copies the base FFN with a different band of its SVD spectrum gently amplified.
- This is meant to break symmetry while preserving most of the original weights.
- The author says more than 92% of the original weights are retained.
Router initialization
- Router weights are initialized from activation cluster centroids derived from legal, STEM, and general Korean text.
- The shared direction is removed first so the router starts with domain awareness.
The author is explicit that this is a hobby project, AI-assisted, and unlikely to beat the original base model broadly, though it may help in some niche domains.
More from Models
- Rumor claims Kimi faked performance by serving Claude; DeepSeek new model surprises in evals — realsohamparekh · 2026-09-11
- GPT-5.6 writes well but is instantly forgettable, user complains — BasedRaddka · 2026-09-11
- Opus Refuses Protein Research Codebase Over 'Safety' Concerns, Dev Considers Rolling His Own — josephdviviano · 2026-09-11
- User Hails Unconfirmed 'DeepSeek 4.1 Flash' as an Inflection Point in LLMs — himanshustwts · 2026-09-11
- Terminal Bench v4: GLM-5.3 Leads at 41.9%, Kimi-K3 Underwhelms at 12.6% — Ok_Warning2146 · 2026-09-11
- GPT-6 Astra beats Factorio with enemies in 44 in-game hours at ~$4,500 API cost — liminal_bardo · 2026-09-11