Google paper: 55-70% of quantized LLM cold-start latency is just model loading
rohanpaul_ai · x · 2026-09-24
A new Google paper finds that for small quantized LLMs on serverless CPUs, 55-70% of cold-start latency comes from model loading — moving weights into memory, not token generation. Bumping Cloud Run memory from 4GB to 8GB unlocks 2× CPU and nearly halves warm inference time.
Related event: Google Paper: 55-70% of Quantized LLM Cold-Start Latency Is Weight Loading(2 posts)→
More from Infra
- UK Sovereign AI invests in Fractile to build next-gen frontier AI chips — charlieharris01 · 2026-11-11
- What happened when I moved vector retrieval into Snowflake SQL instead of orchestrating RAG outside — dima806_dima · 2026-11-11
- AI chip startup Fractile raises $720M Series C for DRAM-on-processor design — Tom_Westgarth15 · 2026-11-11
- Mobile AI Factory: agentic pipeline builds on-device SLMs that beat 5-shot frontier models — kalyan_kpl · 2026-11-11
- Vivo's Starlink direct-to-cell service hits 10M users in two weeks across Brazil — XFreeze · 2026-11-11
- Nan Ya Plastics to spend NT$11.53B on fifth glass fabric plant on AI server demand — zephyr_z9 · 2026-11-11