Sparse MoEs make SSD the last bottleneck: expert streaming breaks the VRAM wall
rickasaurus · x · 2026-11-22
Frontier open models are ultra-sparse MoEs (Qwen 3.8 Flash Next 180B/6B active, GLM 5.3 753B/40B, Kimi K3.1 Pro 2.8T/104B, 3-6% active per token), so keeping everything in VRAM no longer makes sense. The post traces local inference evolution: layer offloading (2023-2025), llama.cpp's --moe-cache RAM expert caching (Oct 2026), and now expert streaming with direct NVMe reads — limited only by SSD. Companion open-source engine Magnitude auto-tunes kernels per hardware, claiming 2x faster than Ollama across Apple Silicon/NVIDIA/AMD/Intel/CPU (8k GitHub stars).
More from Infra
- Nemotron 3.6 Ac Academia Support Lands in llama.cpp: Looped Hybrid with Gated Delta Product — maternion · 2026-11-22
- Inkling 2 Mini on 2× RTX 3090: 557K Context + Vision at ~100 tok/s via NInfer-all — Sisuuu · 2026-11-22
- OpenAI engineer confirms GPT-Live cache affinity bug, fix rolling out Monday — pbbakkum · 2026-11-22
- Lambda runs Nvidia CPO switches on 10,000+ GPU GB300 NVL72 cluster, cutting latency up to ~10% — Beth_Kindig · 2026-11-22
- Dragos planted fake AWS keys in 2M AI prompts — the keys got used — serendip-ml · 2026-11-22
- Running Inkling 2 Mini locally on Mac: 20GB model, ~16 tok/s on M4 Pro — jacob_posel · 2026-11-22