llama.cpp merges PR #30966: run missed MoE experts on CPU instead of PCIe transfers
QuixiAI · x · 2026-11-08
llama.cpp merged PR #30966 on Nov 7, a major software-only upgrade for running large MoE models that don't fit in VRAM — no new GPU required.
- Previously, a MoE cache miss meant uploading the missing expert's weights from system RAM to VRAM over PCIe, painful on eGPU setups
- Now a configurable share of missed experts can execute directly on the CPU via --moe-cache-cpu-share
- A tuning script (scripts/moe-cache-tune.py) auto-benchmarks configs; example output shows throughput rising from 24.2 t/s to 47.6 t/s with share 0.95
Related event: llama.cpp PR runs uncached MoE experts on CPU, boosting low-VRAM inference(2 posts)→
More from Infra
- tszzl Calls for Total Chip and AI Export Ban on China, Allied Compliance Required — tszzl · 2026-11-08
- nanoGPTJAX Author Plans Hardware-Specific Kernels and an NVFP4 Showcase — A_K_Nain · 2026-11-08
- Zyphra's ZHA sparse attention keeps 35B model quality while reading just 1-6% of KV-cache — burny_tech · 2026-11-08
- DGX Spark prices are starting to come down — AIFlow_ML · 2026-11-08
- MOSS-Image 5B runs fully on a phone NPU: ~30s for 512x768 on Snapdragon 8+ Gen 1 — shirosoro · 2026-11-08
- Open-weight models now handle 72% of OpenRouter tokens: build your fallback before prices bite — shashib · 2026-11-08