Sleeping models in Rust: reclaiming 79GB of idle VRAM with sub-200ms wake
ahstanin · reddit · 2026-09-12
Running a local multi-modal agent stack (Qwen3.8-27B LLM + Nemotron STT + Chatterbox TTS + OCR) with all runtimes hot consumed 122GB of idle VRAM, largely because runtimes like SGLang statically reserve big KV pools (often 85% of available memory).
The author built a small Rust controller that puts models to sleep between turns: it flushes execution buffers and KV pools while keeping the process and base weights mapped, avoiding cold restarts.
- LLM idle footprint drops from 87,443 to 39,092 MiB (matching the 27B FP8 weights); STT to 267 MiB; TTS to 3,127 MiB; OCR to 422 MiB
- Total reclaimed: 79,459 MiB (80GB)
- Wake-up is sub-200ms for every model, since no disk read or graph rebuild is needed
The post ends by asking how others handle multi-model memory allocation when instant voice/tool responses matter but a second GPU is off the table.
Related event: Rust Model Sleep Cuts Local Agent VRAM from 122GB to 43GB(2 posts)→
More from coding & agent
- Current AI governance frameworks ignore multi-agent risks like the HuggingFace incident — Miles_Brundage · 2026-09-12
- Hazel team builds AI tool that turns Git repo activity into weekly digests — neurocy · 2026-09-12
- 404 Game Jam: build a playable 3D game from one prompt, win 10 TAO — bittingthembits · 2026-09-12
- Robotics researcher: SAM plus a light LM already handles most pick-and-place — chris_j_paxton · 2026-09-12
- Open-Source Life Recorder: 24/7 iPhone Audio to Local Whisper Feeds Agents — TheMoonMidas · 2026-09-12
- SurrealDB 3.3.0 Beta Partners with GradiumAI for Voice Agents with Memory — mattturck · 2026-09-12