MoE inference: active parameters matter more than total size
giffmana · x · 2026-07-20
A reply arguing that with MoE, SSD streaming makes local LLM inference feasible even at batch size 1, so the total parameter count matters less than the number of active parameters.
The key claim is that reducing active parameters is always a win, and the remaining tradeoff is mainly that matmul efficiency can become poor in this setup.
Related event: MoE and SSD Streaming Redefine Inference Parameters(3 posts)→
More from Infra
- Sail equivalence proof and riscv-coq cross-checks uncover real RISC-V spec bugs — xennygrimmato_ · 2026-10-27
- xAI's Colossus 1 hits ~14.6 exaflops FP64, ~5x Top500 leader — cis_female · 2026-10-27
- Lambda runs 19 B200 nodes in a 16-node power budget, boosting throughput 24% — TheZachMueller · 2026-10-27
- Podcast: Micron's HBM roadmap, Meta's AI infra, and NVIDIA's supply chain lock-in — BenBajarin · 2026-10-27
- SGLang MegaKernel + PD Pushes the TCO-Latency Frontier on GB300, SemiAnalysis Benchmarks Show — AccBalanced · 2026-10-27
- Scaling self-hosted Convex: compiling TypeScript Convex functions to WASM — leventov · 2026-10-27