Sleeping models in Rust: reclaiming 79GB of idle VRAM with sub-200ms wake
ahstanin · reddit · 2026-09-12
Running a local multi-modal agent stack (Qwen3.8-27B LLM + Nemotron STT + Chatterbox TTS + OCR) with all runtimes hot consumed 122GB of idle VRAM, largely because runtimes like SGLang statically reserve big KV pools (often 85% of available memory).
The author built a small Rust controller that puts models to sleep between turns: it flushes execution buffers and KV pools while keeping the process and base weights mapped, avoiding cold restarts.
- LLM idle footprint drops from 87,443 to 39,092 MiB (matching the 27B FP8 weights); STT to 267 MiB; TTS to 3,127 MiB; OCR to 422 MiB
- Total reclaimed: 79,459 MiB (80GB)
- Wake-up is sub-200ms for every model, since no disk read or graph rebuild is needed
The post ends by asking how others handle multi-model memory allocation when instant voice/tool responses matter but a second GPU is off the table.
Related event: Rust Daemon Sleeps Idle Models, Cutting VRAM from 122GB to 43GB(3 posts)→
More from coding & agent
- HABIT-Bench: Best agent memory system scores just 33.61% at inferring user habits — SJTU · 2026-10-28
- Memory Tax study: dedicated agent memory systems often lose to naive RAG — UIUC-CS · 2026-10-28
- Regehr's 'Piano Test': would you trust formally verified code with your life? — moyix · 2026-10-28
- Vibe-coded 'AI 浮生记' game satirizes AI startup life with compute black-market trading — vista8 · 2026-10-28
- Cursor ships Annotate in design mode: agents take visual prompts via on-page markup — soleio · 2026-10-28
- The legendary fake Go generics: Unicode Canadian Syllabics as angle brackets — rickasaurus · 2026-10-28