GKE Pod snapshots cut AI inference cold starts by 89%, loading 70B models in 37s
rseroter · x · 2026-09-22
- Google Cloud introduced GKE Pod snapshots, a feature that saves a workload's running state (including CPU and GPU memory) and restores it on demand instead of cold-starting.
- Cold start reduced by up to 89%: 70B parameter models load in just 37 seconds, 8B models in 15 seconds.
- Pain point: inference servers must download gigabytes of weights into GPU memory (often minutes), and agentic workloads like code execution sandboxes need fast, per-request isolated environments — startup latency hurts UX and prevents rapid autoscaling, forcing overprovisioning.
- Resuming rather than restarting lets infrastructure scale with demand and cuts overprovisioning costs.
More from Infra
- KubeCon keynote: computing is shifting from human speed to agent speed — HowDevelop · 2026-11-12
- Leaked memo: Meta infrastructure head Santosh Janardhan to leave in early 2027 — dinabass · 2026-11-12
- Omarchy 4.6 lets you share local AI models across your tailnet, says DHH — alexcovo_eth · 2026-11-12
- Eric Schmidt's foundation funds French NGO fighting against data centers — StephanSturges · 2026-11-12
- Epoch: Open-Weight GLM 5.3 Hacks Six Firefox Versions for ~$1,000 Each; Claude Fable 5.2 Sets ECI Record — Epoch AI · 2026-11-12
- Polymarket prices 28% odds of a state data center moratorium; Torus raises $500M — Polymarket · 2026-11-12