Multi-model on one node: 79GB idle GPU memory reclaimed with sub-200ms warmups
ahstanin · reddit · 2026-09-12
A Reddit writeup on single-node multi-model serving: running a 27B LLM (FP8) alongside STT, TTS and OCR models idled at 122GB GPU memory, mostly SGLang KV-cache preallocation and static buffers. Since audio/vision models are mutually exclusive and the LLM sits idle during tool calls, the author built a Rust daemon that flushes dynamic memory on idle while keeping process residency.
Results:
- LLM idle memory: 87,443 → 39,092 MiB; overall 79.4GB reclaimed
- STT: 10,385 → 267 MiB; TTS: 17,947 → 3,127 MiB; OCR: 6,592 → 422 MiB
- Sub-200ms wake latency by avoiding disk I/O and graph reconstruction
- SGLang settles at 39GB at rest, matching model weights
Related event: Rust Daemon Sleeps Idle Models, Cutting VRAM from 122GB to 43GB(3 posts)→
More from Infra
- MySQL recursive CTEs time out at 500M rows; C# BFS answers in 2.7ms vs NebulaGraph 23.6ms — JeremyCMorgan · 2026-10-28
- Token demand on OpenRouter grew 27x to 165T weekly, open models now serve 72% — FinanceYF5 · 2026-10-28
- AI's unusual bet: buying compute for years, selling intelligence by the token — vaibhavbetter · 2026-10-28
- OpenAI's Will Depue says compute locks in US primacy as 'end of human history' nears — willdepue · 2026-10-28
- Malaysia has overtaken Singapore in data center capacity, planned buildout 10x — bytebot · 2026-10-28
- Why Cloudflare Is the Best Platform to Build AI Agent Operating Systems — ritakozlov · 2026-10-28