vLLM-Metal merges sleep mode: frees KV cache memory on idle Macs without restart
abhijithneil · x · 2026-10-15
A PR authored by abhijithneil was merged into vLLM-Metal, bringing level 1 sleep mode to Apple Silicon Macs. An idle vLLM engine keeps its KV cache pool allocated, which on a 16GB Mac mini can force swapping. With --enable-sleep-mode, sleep()/wakeup() now release and restore KV cache memory in a fraction of a second while keeping model weights loaded — no engine restart needed. KVCacheStorage gained release()/reallocate() with identical layout so scheduler block tables stay valid. Ideal for background model servers that mostly sit idle between requests.
More from Infra
- Decision Models mining to support smaller models, even CPU-only like Laya — airesearch12 · 2026-10-15
- Tree packing lands in TRL AsyncGRPO, cutting agentic RL trainer compute 26x — SergioPaniego · 2026-10-15
- JSSA sparse attention hits 6.43x decoding throughput at 128K context on B200 GPUs — SNU-VLSI · 2026-10-15
- 189 benchmarks later: how to pick KV quants for Qwen3 27B on 16GB VRAM — Fancy-Snow7 · 2026-10-15
- Energy, compute, data are the only new currency: $15B Anthropic-SpaceX deal caps 22 days of AI shakeups — sierracatalina · 2026-10-15
- Titan RTX + 3090 without NVLink: is multi-GPU inference via Strata worth it? — Express_Quail_1493 · 2026-10-15