Tinkering: Attempting to Extract MoE Experts from Kimi K3
StableDiffer · reddit · 2026-07-30
A developer experimented with using REAP (routing-based expert removal) to extract individual expert networks from Mixture-of-Experts (MoE) models like Kimi K3 or GLM 5.2.
Due to hardware constraints, the author could only run a single K3 expert. They wonder whether these extracted experts might exhibit general utility for different specific tasks. While the idea of compressing the model into a 104B parameter version is intriguing, the author admits that such a crudely extracted model would likely perform poorly in practice.
More from Models
- First Impressions: Gemini 4 Argon 'Incredible Writer' Beats Opus 5.5 and GPT 6 Astra — soumitrashukla9 · 2026-10-16
- Trying Grok bot surfaces an unpaid Cursor balance, blocking account merging — haydendevs · 2026-10-16
- TensorFold 1.0.12 runs 120B vision model on 1-2 DGX Sparks, adds Mac image input — HankYeomans · 2026-10-16
- Leaked screenshot shows unannounced Gemini 4 Argon gated to Ultra and paid Pro subscribers — il_-bro · 2026-10-16
- Google: Gemini 4 Argon lands in Antigravity, code submissions up 35%, rollbacks down ~33% — tekbog · 2026-10-16
- Google's "ungrounded beliefs" classifier injections derail Opus 4.1's reasoning chain — liminal_bardo · 2026-10-16