Sparse MoEs make SSD the last bottleneck: expert streaming breaks the VRAM wall

rickasaurus · x · 2026-11-22

Frontier open models are ultra-sparse MoEs (Qwen 3.8 Flash Next 180B/6B active, GLM 5.3 753B/40B, Kimi K3.1 Pro 2.8T/104B, 3-6% active per token), so keeping everything in VRAM no longer makes sense. The post traces local inference evolution: layer offloading (2023-2025), llama.cpp's --moe-cache RAM expert caching (Oct 2026), and now expert streaming with direct NVMe reads — limited only by SSD. Companion open-source engine Magnitude auto-tunes kernels per hardware, claiming 2x faster than Ollama across Apple Silicon/NVIDIA/AMD/Intel/CPU (8k GitHub stars).

Original post →

More from Infra

Infra channel →