vLLM gains sleep mode on Apple Silicon: idle model servers now return KV cache memory
abhijithneil · x · 2026-10-21
Abhigith Neil Abraham contributed a sleep mode feature to vLLM's Apple Metal backend, with his walkthrough now featured on the official vLLM Metal site.
The problem: a small model server kept running all day on a Mac holds its KV cache pool even when idle. Under Apple Silicon's unified memory, this competes with other apps and causes swapping.
The fix: launch with --enable-sleep-mode, then call sleep() to return KV cache memory to the system and wakeup() to reclaim it in a fraction of a second. The post explains vLLM's block-based KV pool, why idle engines hold useless prefix cache, and how weights are retained while cache is released.
More from coding & agent
- Research agent Épi sets 120 records, narrows n-queens constant interval 84x — dair_ai · 2026-10-21
- Compaction hooks in practice: keep last n messages verbatim, summarize the rest for full agent continuity — MoonL88537 · 2026-10-21
- The blank prompt box hides real operational burden: context, workflow and supervision fall on users — _AustinCalvert_ · 2026-10-21
- DeepMind paper: self-evolving agents forget as they learn — fastest learner GEPA also breaks old tasks most — rohanpaul_ai · 2026-10-21
- arXiv: Are Self-Evolving Agents Continual Learners? Plasticity comes at the cost of stability — rohanpaul_ai · 2026-10-21
- PM OS vs Team OS vs Company OS: how to layer your AI setup and which to build first — aakashgupta · 2026-10-21