Tinkering: Attempting to Extract MoE Experts from Kimi K3
StableDiffer · reddit · 2026-07-30
A developer experimented with using REAP (routing-based expert removal) to extract individual expert networks from Mixture-of-Experts (MoE) models like Kimi K3 or GLM 5.2.
Due to hardware constraints, the author could only run a single K3 expert. They wonder whether these extracted experts might exhibit general utility for different specific tasks. While the idea of compressing the model into a 104B parameter version is intriguing, the author admits that such a crudely extracted model would likely perform poorly in practice.
More from Models
- GPT-5.6 Sol Reasoning Details: Lack of Memory Forces Re-learning Every Step — charliermarsh · 2026-07-30
- User Complains Kimi Credits Evaporate Too Fast, Demands $200/Mo Subscriptions — doodlestein · 2026-07-30
- FAR AI Security Leaderboard: Some Models Jailbroken for Under $300 — AndyMasley · 2026-07-30
- Anthropic CEO: AI Model Finds 271 Firefox Vulnerabilities, Prioritizing Defenders — firasd · 2026-07-30
- Anthropic Accused of Posting Misleading Benchmark Numbers in Victory Tweet — soumitrashukla9 · 2026-07-30
- GPT-6 Rumored to Undergo New Pre-training Run, Potentially Yielding Massive Leap — haider1 · 2026-07-30