llama.cpp user asks whether MoE experts can be streamed from SSD instead of RAM
Ok_Warning2146 · reddit · 2026-07-23
Can llama.cpp offload non-shared MoE experts to SSD?
The poster asks whether llama.cpp can go beyond its current -cmoe setup, which loads non-shared experts of MoE models into RAM while keeping other weights and KV cache on GPU VRAM.
The question
They want to know if the same idea can be extended so that:
- non-shared experts live on SSD instead of RAM,
- CPU handles execution,
- and the rest of the model remains on GPU VRAM.
Why it matters
The motivation is simple: RAM is expensive, and MoE models keep getting larger. The question is essentially about more aggressive tiered memory management for local inference.
More from coding & agent
- Open-source WeChat Channels download Skill lets users save videos by URL — vista8 · 2026-07-23
- Fleet engineering turns vibe coding into parallel AI coding sessions — victor_explore · 2026-07-23
- Open repo packages 21 chapters of agentic design patterns with code notebooks — tom_doerr · 2026-07-23
- Keel Code launches with a loop of frontier models that keeps coding agents going until the goal is done — Scobleizer · 2026-07-23
- Stas Bekman promotes another SKILL.md for his AI toolkit — StasBekman · 2026-07-23
- Databricks demos moving from template to deployed app with Replit’s agent — amasad · 2026-07-23