llama.cpp MoE expert LRU cache boosts RTX 5090 local inference 2.2x to 68 tok/s
abhishekkumar333 · reddit · 2026-10-22
llama.cpp PR #29887 (merged Oct 7, ported from Tether's qvac-fabric) adds an LRU cache of MoE experts in VRAM — possibly the most impactful local-inference change this year.
Why it works: expert routing has locality of reference — consecutive tokens mostly pick the same hot experts. On a hit, zero bytes move; on a miss, one 2.6 MiB expert is copied over PCIe and the LRU victim is evicted. For Qwen3.8-Flash-Next (125B total, 6B active), each token touches 1.3 GB of expert weights.
Benchmarks (same model, same machine): RTX 4090 24GB goes 25.0 → 40.7 tok/s (1.62x, 77% hit rate); RTX 5090 32GB goes 30.8 → 67.8 tok/s (2.20x, 89%); AMD R9700 32GB gains 2.5x at 94.6% hit rate.
Usage: llama-server ... -cmoe --moe-cache-mib 11000; rule of thumb is 10% of expert bytes. SGLang landed the same idea 12 days later as "paged experts" (PR #42557). The write-up includes a memory-math script and a runnable expert-cache simulator.
More from Infra
- GitHub moves 99.99999% of repos to Azure; per-reference quorums fix 25ms latency wall — rseroter · 2026-10-22
- Flint model weights hit HuggingFace, runs on DGX Spark, RTX GPUs and M4 Pro — testingcatalog · 2026-10-22
- ABB ships RobotStudio HyperReality on NVIDIA Omniverse; Absolute Accuracy calibration is key — lukas_m_ziegler · 2026-10-22
- Flappy Airplanes raises $1B Series A co-led by Index and Lux for new GPU primitives — gabriel1 · 2026-10-22
- Researcher: the final megakernel isn't a kernel but a transformer in an endless loop — DimitrisPapail · 2026-10-22
- SPCX earnings call highlights 'computer per watt' metric as AI value shifts down the stack — mobileraj · 2026-10-22