Dev converts Qwen3-0.6B into a MoE model: quality drops and no speed gain at this scale
Bayliner1980 · reddit · 2026-10-23
Reddit user bayliner1980 continues his MoE-fication project, this time converting Qwen/Qwen3-0.6B into a sparse MoE (0.5B active) with weights and GGUF files on Hugging Face.
Key technical notes: Qwen3 lacks a shared expert, which hurt quality after conversion, so he introduced a "pseudo-shared expert" (an expert the router always picks) to recover some of it; GGUF compatibility forced further compromises. Results: Wiki perplexity rises from 23.88 to 26.98, all task metrics fall below the dense original; GPU generation actually slows slightly (377→352 t/s) while CPU generation improves marginally (115→124 t/s)—at 0.6B scale there's no real speed win, unlike his earlier 1.5B conversion.
Honest limitations: confident factual errors (says da Vinci painted the Mona Lisa in 1519, the year he died), broken code, non-thinking mode only. He's also soliciting feedback on a neuron-duplication experiment that improves quality but doubles parameter count.
More from Models
- Gemini somehow can't read its own Personal Instructions, user's toggles don't help — Connect_Reporter_799 · 2026-10-23
- Diffusion LLM bias rankings shift with sampling configs, persist after debiasing — Bollegala · 2026-10-23
- Running ~100 DeepSeek v4.1 flash agents in parallel for under $20/month — zack_overflow · 2026-10-23
- Researcher laments open-source audio models are surprisingly subpar, even representation ones — gowthami_s · 2026-10-23
- User claims they generated video with "Claude Opus 5.5", unannounced model name surfaces — jwt0625 · 2026-10-23
- Security researchers push back as OpenAI restricts Astra access; BlackHat 2027 mocked as 'cyber model' bazaar — cyb3rops · 2026-10-23