llama.cpp hits v1 with MoE offloading and LRU cache for VRAM-bound models
danielhanchen · x · 2026-11-14
llama.cpp officially released v1, with new MoE offloading plus smart LRU caching so large MoE models that don't fit in VRAM can still run fast. Enable it with --moe-cache-mib N to dedicate a set amount of GPU VRAM (in MiB) to the MoE cache; Unsloth's prebuilt llama.cpp adds an "auto" option that sizes the cache automatically. A significant memory optimization for local MoE deployment.
More from Infra
- LexiSuite: a Rust-built inference engine beats llama.cpp 1.0.0 at 262k context — W61k3r · 2026-11-14
- Korea Chip Exports Hold Near $2.2B/Day as DRAM Prices Jump 5.5% — Supply Wall or Demand Crack? — tengyanAI · 2026-11-14
- GPU forensics suggest DeepSeek's real compute far exceeds disclosed figures — teortaxesTex · 2026-11-14
- NVIDIA allegedly funded AI startups to buy its GPUs, engineering scarcity — dr1337 · 2026-11-14
- Smuggling GB300s labeled as "timber": AI circle's chip-smuggling pirate meme goes viral — tszzl · 2026-11-14
- Investor argues AI stocks are a "nubble," not a bubble—growth is real, pricing isn't — JOBhakdi · 2026-11-14