Google paper: model loading takes 55-70% of cold-start latency for quantized SLMs on serverless CPUs
rohanpaul_ai · x · 2026-09-24
A new Google research paper systematically benchmarks quantized small language models served via llama.cpp on Cloud Run's CPU-only infrastructure.
Key findings:
- Model loading dominates cold-start time at 55-70%; the bottleneck is moving weights into memory, not token generation
- The 8 GiB memory tier unlocks a hidden 2× vCPU advantage, nearly halving warm inference time
- Q4KM quantization achieves a Pareto-optimal tradeoff across latency, throughput, and quality (validated via WikiText-2 perplexity)
- A three-predictor latency model (R²=0.97) and break-even cost analysis offer a quantitative framework for deployment decisions
Scope: 5 models (270M-3.8B), 4/8 GiB memory tiers, and a Q2K through Q80 GGUF quantization sweep. Practical guidance for anyone deploying SLMs on serverless platforms.
Related event: Google Paper: 55-70% of Quantized LLM Cold-Start Latency Is Weight Loading(2 posts)→
More from Infra
- Gas-for-AI-datacenters forum co-located with CoRL shows energy demand spillover — carlosdponx · 2026-11-12
- a16z podcast: Mike Volpi on Cisco's 70+ acquisition playbook and the AI buildout — a16z · 2026-11-12
- AMD to pass Intel in server revenue by mid-2027 as share gains 6 points a year — BenBajarin · 2026-11-12
- Study: Reflective Data Center Roofs Actually Cool Daytime Ground by 1.8-2.2°C — SumitGup · 2026-11-12
- Tsinghua's BaGuaLu-2 tames MoE expert-load spikes at runtime, accepted to SC26 — jiqizhixin · 2026-11-12
- antirez backs low-bit DeepSeek 4.1 Flash over Qwen 27B dense on Apple M5 Macs — antirez · 2026-11-12