MoE inference: active parameters matter more than total size
giffmana · x · 2026-07-20
A reply arguing that with MoE, SSD streaming makes local LLM inference feasible even at batch size 1, so the total parameter count matters less than the number of active parameters.
The key claim is that reducing active parameters is always a win, and the remaining tradeoff is mainly that matmul efficiency can become poor in this setup.
Related event: MoE and SSD Streaming Redefine Inference Parameters(3 posts)→
More from Infra
- Anthropic locked into $475B compute deals, OpenAI lifts projected spend to ~$750B through 2030 — kyliebytes · 2026-10-27
- First systematic data center noise study: 17 of 25 hyperscalers drew no credible complaints — AndyMasley · 2026-10-27
- Fork builds engine to run Qwen at ~400 tok/s on three used V100s — jfinkpottery · 2026-10-27
- RFD4-Proteina runs on an RTX 3060: protein design from $0.001 per design — AllThingsApx · 2026-10-27
- NPM packages now take ~20 minutes to go live, dev suspects AI-attack throttling — cocktailpeanut · 2026-10-27
- Nature Physics: reprogrammable refractive index enables arbitrary multimode wave control for ML — MarioKrenn6240 · 2026-10-27