Cloudflare Details Inference Optimizations for Running Kimi and GLM at Scale
michellechen · x · 2026-08-03
Cloudflare published a technical blog sharing underlying optimization experiences for running trillion-parameter models like Kimi and GLM at scale on their Workers AI platform.
To solve memory constraints for these large, long-context MoE models, the team applied three main techniques:
- Quantizing the KV cache: Effectively reduces GPU memory usage from attention keys and values.
- Compressing model weights: Further lowers memory requirements for model loading.
- Protecting the cache: Prevents cache interference when packing more requests onto shared hardware.
Additionally, all experiments and production traffic run and benchmark on the open-source inference framework SGLang, which the team found offers the best performance on the market.
More from Infra
- CXL Disaggregated Memory Nears Commercial Deployment, Says Ben Bajarin After FMS Meetings — BenBajarin · 2026-09-23
- Sandboxing ComfyUI for AI agents in a VM with a single shared GPU remains unsolved on Windows — johnshedletsky · 2026-09-23
- Unsloth GGUFs run Qwen-Image-2.1 locally on as little as 6GB VRAM — danielhanchen · 2026-09-23
- DigitalOcean launches Managed Agents with 16,000+ tools, Claude Code and Codex support — dair_ai · 2026-09-23
- H Company trains computer-use agents on SkyPilot: thousands of sub-second sandboxes — skypilot_org · 2026-09-23
- MiniMax H3 video gen runs locally on M5 Ultra: 768p in ~2m22s with optimizations — bakawolf123 · 2026-09-22