Under a total memory constraint, unactivated MoE params are dead weight
TimDarcet · x · 2026-10-21
The author argues that under a total memory budget, MoE parameters that stay unactivated are pure dead weight on VRAM. Implication: for the best intelligence per GB, you might as well go dense. He also praises the Llama 4 tokenizer for just working without headaches.
Related event: Opinion: Under Memory Constraints, Inactive MoE Parameters Are Dead Weight(2 posts)→
More from Models
- ChatGPT desktop app crashing every few minutes, killing users' agents — wstone_bd · 2026-10-21
- Polish dev defends open model: built on Gemma with more than a simple LoRA fine-tune — ColinWPLewis · 2026-10-21
- X trends: SophontAI's Holon-S1 predicts gene activity from cancer slides; decision models rise — iScienceLuvr · 2026-10-21
- OpenAI's MultiLangBench: PHP scores highest (91.3%) at building a database engine from scratch — ricklamers · 2026-10-21
- 704-vote robotics Elo ranks Claude Opus 5.5 first at 1379; no model completes the task — qinzytech · 2026-10-21
- DeepSeek V4 called 'god-tier infra, 4th-tier ML': 7 tricks stacked, 20 patches to fix them — teortaxesTex · 2026-10-21