Developer upcycles Gemma2-4B into a 4-expert MoE model
Desperate-Sir-5088 · reddit · 2026-09-01
A developer shared results from upcycling the Gemma2-4B dense model into a Mixture-of-Experts (MoE) architecture. By adding 4 experts, the model's abilities were confirmed to recover to a 'general level'. The developer also called on Google to release the official 124B MoE model.
More from Models
- DeepSeek V4.1 Flash enters BuildingBench top five at 1/15 of Sonnet's cost — Lianhuiq · 2026-10-18
- MOSS SFT vs Flux 3 Klein vs SD 3.0: Same-Prompt Comparison Exposes SD3's Age — ZootAllures9111 · 2026-10-18
- Quick take: GPT-6 matches Opus 5.5 at coding, far ahead at computer use — _vztu · 2026-10-18
- Open models now above Sol-level, outpacing expectations as community awaits V4.1 Pro — yacineMTB · 2026-10-18
- Codex took 3 days, Opus 5.5 nailed it in an afternoon — banned user's video protest — dotey · 2026-10-18
- Yacine MTB: Users will remember how Anthropic treated them once intelligence proliferates — yacineMTB · 2026-10-18