Offloading only "hot" MoE experts to VRAM boosts llama.cpp throughput 50%
nbvehrfr · reddit · 2026-08-29
A developer forked llama.cpp with a twist: when a MoE model doesn't fit entirely in VRAM (here Qwen 3.8 Flash Next), instead of offloading whole layers, only the "hot" experts go to VRAM—raising throughput from 20 t/s to 30 t/s (+50%).
Key observation: certain groups of experts stay relatively stable across coding, refactoring and code-review workloads, making them worth pinning. PR: github.com/timadinorth/llama.cpp/pull/1.
Caveats: tested only on coding workloads, and only useful when the full model exceeds VRAM. The author doubts upstream will accept it—the low-level implementation was written by Opus, and he doesn't feel like defending every line.
More from Infra
- Observation: Why Are So Many People Suddenly Owning DGX Stations? — andrew_n_carr · 2026-08-29
- Understanding KV, Prefix, Prompt, and Semantic Caching in LLMs — blaizedsouza · 2026-08-29
- Tenstorrent Quietbox 2 Arrives: 256G RAM, 128G Interconnected GDDR for the Price — SashaUsesReddit · 2026-08-29
- DeepSeek V4 pricing outruns GLM-5.3 and Qwen-3.8 in cost-performance debate — teortaxesTex · 2026-08-29
- Poor EDA tool file compression creates SaaS opportunity — ai · 2026-08-29
- Unions support data centers, highlighting benefits over dismissal — apples_jimmy · 2026-08-29