Hot-swapping 18GB models RAM↔VRAM on one 24GB GPU: 1.5s physics vs 29s llama.cpp reality
bjivanovich · reddit · 2026-10-31
A Reddit user running a sequential multi-agent pipeline (Architect→Coder→Debugger) on a single RTX 3090 profiles why swapping two 14-18GB models takes 25-30.5s in llama.cpp when the PCIe transfer should only take 1.5s.
Why it's slow
- llama-server router mode kills and respawns the child process per model switch
- Each new process pays cold CUDA context init (cuInit, context, CUBLAS handles): 3-5s
- Empty page tables trigger millions of soft page faults touching 15+GB of GGUF on Windows
- Re-allocating the 24GB VRAM pool and KV cache buffers takes seconds more
What's needed
- One persistent process with a warm CUDA context
- Idle model kept in page-locked pinned host memory (cudaHostAlloc / VirtualLock)
- Direct async DMA copy (cudaMemcpyAsync) into a pre-allocated GPU buffer, then repoint graph pointers
The author asks whether Strata, NInfer, or custom vLLM/PyTorch memory-pool forks can hot-swap weights across architectures in <2s, and whether llama.cpp has a PR that retains VRAM allocation and only reloads weights.
More from coding & agent
- 7 free Claude Skills that fix the cookie-cutter look of AI-generated pages — socialwithaayan · 2026-10-31
- Tibo's 28 Days of Shipping: 8 Resets in 28 Days Reveals AI Coding Tool Instability — heypearlai · 2026-10-31
- Developer ships one AI-built micro app per day with Opus 5.5, launches a full collection on TikTok in under a month — ilumine_ai · 2026-10-31
- Codex now lets you take a photo mid-session on mobile — Dimillian · 2026-10-31
- Retro funding redesign: prediction markets cut reviewer load by only judging challenged projects — devanshmehta · 2026-10-31
- Using dot, Grok bot, or Claude Duet to watch a cluster is a surprisingly effective SRE strategy — sloppenheimer · 2026-10-31