SGLang's Weight Cache Daemon Cuts 1T Model Restart Time from 8.8 min to 32 sec
xiaosun86 · x · 2026-08-22
SGLang, in collaboration with Ant Group and Alibaba, introduced the Weight Cache Daemon to drastically reduce model reload times. By keeping post-quantized weights persistent in GPU memory and using CUDA IPC zero-copy mapping, the new system achieves:
- Weight Loading: 495s → 0.63s (approx. 785x speedup) on a Ling-2.6-1T FP8 model.
- Total Startup: 8.8min → 32s (93.9% reduction).
Key Features:
- Zero-Copy Loading: New engine instances map directly to existing GPU memory.
- Multi-Instance Sharing: Eliminates redundant disk I/O and transforms for multiple instances on the same GPU.
- Active-Standby Failover: Enables sub-second failover without dedicating full GPU memory to standby engines.
This is the first phase of their Fast Engine Recovery Framework, targeting <10s cold restarts and <1s warm standby switches for production LLM serving.
More from Infra
- Analyst: 80% of neoclouds may not survive, but the demand for their GPU capacity is real — BenBajarin · 2026-10-16
- Apple and Meta partner to run Muse Spark model on linked Mac Studios — pdamodaran · 2026-10-16
- Nvidia's developer base hits 11 million, up ~3.7x from 3 million in 2021 — Beth_Kindig · 2026-10-16
- Muse Spark Open runs locally on a Mac M3 Ultra via llama.cpp: frontier model on desktop — pcuenq · 2026-10-16
- Muse-Spark open-weights drop: 816B-A42B MoE model gets 359GB MLX 3-bit build — ns123abc · 2026-10-16
- Meta launches Muse Spark Open, says it runs on two M5 Ultra Mac Studios with Apple's help — The Verge AI · 2026-10-16