Total Parameters No Longer Matter in MoE Inference
askerlee · x · 2026-07-20
In a reply, the author shares an insight regarding inference storage: since MoE allows the number of activated parameters to shrink, weights can be loaded via SSD streaming.
The core argument is that total parameter count is no longer the primary constraint. What truly matters is the number of parameters actually activated per inference. Therefore, reducing active parameters is always beneficial, meaning there are 'no trade-offs'.
Related event: MoE and SSD Streaming Redefine Inference Parameters(3 posts)→
More from Infra
- Anthropic S-1 reveals $800B+ in off-books future compute commitments — GaryMarcus · 2026-10-27
- ~42% of Anthropic's ~11GW contracted compute is NVIDIA GPUs — ns123abc · 2026-10-27
- EdgeRock-4B holds KV cache at 72 MiB across 256K context, 99.1% smaller than Qwen3.5-4B — alifcoder · 2026-10-27
- RTX 5080 price jumps from $1658 to $2318 in ten days on Amazon — Frizzy-MacDrizzle · 2026-10-27
- ComfyUI on 12GB VRAM: measured peaks show offloading turns 24s into 11 minutes — Impossible-Mail-4619 · 2026-10-27
- Neon rebuilds its docs search entirely on its own stack: Lakebase Search, Functions, AI Gateway — matei_zaharia · 2026-10-27