Multi-model on one node: 79GB idle GPU memory reclaimed with sub-200ms warmups
ahstanin · reddit · 2026-09-12
A Reddit writeup on single-node multi-model serving: running a 27B LLM (FP8) alongside STT, TTS and OCR models idled at 122GB GPU memory, mostly SGLang KV-cache preallocation and static buffers. Since audio/vision models are mutually exclusive and the LLM sits idle during tool calls, the author built a Rust daemon that flushes dynamic memory on idle while keeping process residency.
Results:
- LLM idle memory: 87,443 → 39,092 MiB; overall 79.4GB reclaimed
- STT: 10,385 → 267 MiB; TTS: 17,947 → 3,127 MiB; OCR: 6,592 → 422 MiB
- Sub-200ms wake latency by avoiding disk I/O and graph reconstruction
- SGLang settles at 39GB at rest, matching model weights
Related event: Rust Daemon Sleeps Idle Models, Cutting VRAM from 122GB to 43GB(3 posts)→
More from Infra
- OpenAI engineers: AI-found kernel optimizations cut GPT-5.6 Sol serving cost by 20% — TheTuringPost · 2026-09-12
- Polymarket pegs 18% odds of an orbital AI data center by end of 2027 — Polymarket · 2026-09-12
- Ayar Labs Extends Series E by $150M, Bringing Total 2026 Funding to $650M — bookwormengr · 2026-09-12
- Lightning AI opens 35 new roles in New York after Voltage Park merger — LightningAI · 2026-09-12
- Musk announces Terafab: Tesla, SpaceX and xAI to build 1TW/year chip fab — elonmusk · 2026-09-12
- The hidden cost of agents is KV cache: DeepSeek compresses to ~890 bytes per token — altryne · 2026-09-12