llama.cpp v0.7.0 adds GPU LRU cache for host-memory MoE experts, up to 2.2x faster decode
EAccelerate_42 · x · 2026-10-16
llama.cpp v0.7.0 ships --moe-cache-mib, a GPU LRU cache for MoE experts kept in host RAM, delivering up to 2.2x faster decode with expert offload; the cache is shared across multiple GPUs.
Other highlights: LiliCorr speculative decoding (small draft transformer on top of DFlash) and MTP for GLM5-Next; a batch of new models including K2 Horizon, Prism Bonsai 2 27B, EmbeddingGemma-2, LiquidAI d1/d1-omni, MiniCPM-V 4.7, and Nemotron Super 3.5 VL; a refactored chat session API, much better slot and prompt-cache restore, a new default port (9931), a full models manager in the web UI, and ggml updated to v0.27.0.
More from Infra
- New NVFP4 trick: compressing block scales saves 45-48% per expert matrix, bit-exact — TheZachMueller · 2026-10-16
- AWS extends Claude Sonnet 3/3.5/3.6 EOL on Bedrock to November 13, 2026 — repligate · 2026-10-16
- After Apple and Meta CEOs tout Mac Silicon inference, a $10B distributed network idea resurfaces — gajesh · 2026-10-16
- NVIDIA's 74B Nemotron 3.5 Bolt, top US open-weight model, now runs locally on Mac — SumitGup · 2026-10-16
- NVFP4 weight lossless compression frees ~6GB VRAM for KV-cache — TheZachMueller · 2026-10-16
- Meta open-weights 816B Muse Spark Open, Apple tunes it for MLX on Mac — rohanpaul_ai · 2026-10-16