NVIDIA Dynamo Snapshot cuts LLM serving cold-start times by ~10x
Nasereliver · x · 2026-10-01
A SysConf 2026 talk announcement: Rasheedat Atinuke Jamiu will present her experiments on fixing the LLM cold-start problem with NVIDIA Dynamo Snapshot.
Scaling up LLM serving is slow because new GPU workers must initialize the runtime, load model weights, prepare kernels, and allocate memory. Dynamo Snapshot checkpoints an initialized worker and restores new workers from it — using CRIU for host state and cuda-checkpoint for GPU state.
Her experimental results show roughly a 10x drop in startup times.
More from Infra
- Nvidia scandal isn't slowing down: Huang's near-$100B power web under strain — The Verge AI · 2026-11-16
- Volume rules chips: halvarflake says Intel lost by outsourcing demand — TheKanter · 2026-11-16
- Taipei 101 lights up "Taiwan ❤️ Jensen" as tech sector rallies behind Jensen Huang — liu_mingyu · 2026-11-16
- Nous exec: avoid GPT 900k context variant — extra charges kick in past 272k — Teknium · 2026-11-16
- CUDA matmul optimization walkthrough: naive kernel to 93.7% of cuBLAS — RisingSayak · 2026-11-16
- Dev runs 120B models on iOS by streaming MoE weights from flash in llama.cpp — Chance-Resource-4970 · 2026-11-16