llama.cpp PR Caches Hot MoE Experts on GPU, Doubling Inference Speed on 8GB VRAM
BTA_Labs · reddit · 2026-08-05
A new llama.cpp PR (#26563) introduces a heatmap to track the most frequently used MoE experts. Instead of keeping all experts on the GPU or offloading them entirely to the CPU, it caches these "hot" experts in VRAM while cold experts continue running on the CPU.
The author's tests on Qwen3.6-35B-A3B with 8GB VRAM showed significant improvements:
- Q2M: 33.25 → 56.0 tok/s (1.68x speedup)
- Q5KP: 17.34 → 35.93 tok/s (2.07x speedup)
However, this isn't a universal fix: models with low expert reuse like Qwen3.5-122B-A10B were actually slower due to tracking and cache-management overhead. Currently CUDA-only and active only during single-token decoding, this approach offers a promising direction for running large MoE models on consumer GPUs without relying on extreme quantization.
More from Infra
- NSF Launches $100M Program for Regional AI Infrastructure Hubs — mkratsios47 · 2026-08-05
- Chutes AI Enforces TEE Verification: 8x RTX 5090s Beat Pro GPUs at 65% Lower Cost — markjeffrey · 2026-08-05
- engyai Launches Cheapest Kimi K3 API on OpenRouter, Cutting Costs by 50% — const_reborn · 2026-08-05
- LiquidAI's LFM2.5-2.6B Hits 82 tok/s Decode on Mac with 128K Context — helloiamleonie · 2026-08-05
- ai& Partners with Voltaiq for Battery Storage in Japanese AI Data Centers — DavidBennett__ · 2026-08-05
- Best Local LLMs for Coding on a 128GB Mac? — Electronic_Back1502 · 2026-08-05