Google paper: model loading takes 55-70% of cold-start latency for quantized SLMs on serverless CPUs
rohanpaul_ai · x · 2026-09-24
A new Google research paper systematically benchmarks quantized small language models served via llama.cpp on Cloud Run's CPU-only infrastructure.
Key findings:
- Model loading dominates cold-start time at 55-70%; the bottleneck is moving weights into memory, not token generation
- The 8 GiB memory tier unlocks a hidden 2× vCPU advantage, nearly halving warm inference time
- Q4KM quantization achieves a Pareto-optimal tradeoff across latency, throughput, and quality (validated via WikiText-2 perplexity)
- A three-predictor latency model (R²=0.97) and break-even cost analysis offer a quantitative framework for deployment decisions
Scope: 5 models (270M-3.8B), 4/8 GiB memory tiers, and a Q2K through Q80 GGUF quantization sweep. Practical guidance for anyone deploying SLMs on serverless platforms.
Related event: Google Paper: 55-70% of Quantized LLM Cold-Start Latency Is Weight Loading(2 posts)→
More from Infra
- Qualcomm ships early Linux developer preview for Snapdragon X2 series — ryanshrout · 2026-09-24
- DuckDB now ships built-in inside dbt v2 on the Rust-based Fusion engine — josh_wills · 2026-09-24
- AI costs fall ~47% per quarter — 54x faster than electricity, but R&D elasticity is average — RishiBommasani · 2026-09-24
- First Cybercab Built With Locally Made Nickel Cathode From Tesla's Gigafactory Texas — elonmusk · 2026-09-24
- Unverified: GPT-6 Luna Positioned on Pareto Frontier at Pennies Per Task, GPT-6 Sol Cuts Cost — haider1 · 2026-09-24
- Tech Giants Offered Up to $300B in Guarantees for AI Data Centers, Chips: FT — FinanceYF5 · 2026-09-24