Cloudflare's Memory Optimization for Serving Kimi and GLM at Scale
arpit_bhayani · x · 2026-08-05
Cloudflare shared their engineering practices for running large models like Kimi K2.6 and GLM 5.2 at scale. For long-context models, the KV cache, rather than model weights, is typically the first to exhaust GPU memory.
To increase concurrency and reduce costs, Cloudflare applied three core optimizations:
- KV Cache Quantization: Reducing Kimi K2.6's cache from BF16 to FP8 precision, instantly halving its memory footprint.
- Weight Compression: Further compressing model weights to free up space.
- Cache Protection: Ensuring cache safety and stability on shared hardware with increased request loads.
For the underlying inference framework, the team uses and actively contributes to the open-source framework SGLang, citing it as the best performing solution currently on the market.
More from Infra
- CXL Disaggregated Memory Nears Commercial Deployment, Says Ben Bajarin After FMS Meetings — BenBajarin · 2026-09-23
- Sandboxing ComfyUI for AI agents in a VM with a single shared GPU remains unsolved on Windows — johnshedletsky · 2026-09-23
- Unsloth GGUFs run Qwen-Image-2.1 locally on as little as 6GB VRAM — danielhanchen · 2026-09-23
- DigitalOcean launches Managed Agents with 16,000+ tools, Claude Code and Codex support — dair_ai · 2026-09-23
- H Company trains computer-use agents on SkyPilot: thousands of sub-second sandboxes — skypilot_org · 2026-09-23
- MiniMax H3 video gen runs locally on M5 Ultra: 768p in ~2m22s with optimizations — bakawolf123 · 2026-09-22