Cloudflare Details Inference Optimizations for Running Kimi and GLM at Scale
michellechen · x · 2026-08-03
Cloudflare published a technical blog sharing underlying optimization experiences for running trillion-parameter models like Kimi and GLM at scale on their Workers AI platform.
To solve memory constraints for these large, long-context MoE models, the team applied three main techniques:
- Quantizing the KV cache: Effectively reduces GPU memory usage from attention keys and values.
- Compressing model weights: Further lowers memory requirements for model loading.
- Protecting the cache: Prevents cache interference when packing more requests onto shared hardware.
Additionally, all experiments and production traffic run and benchmark on the open-source inference framework SGLang, which the team found offers the best performance on the market.
More from Infra
- Cloudflare Restructures Cloud Agent Architecture: Container Sandboxes On-Demand — irvinebroque · 2026-08-03
- Running MiniMax H3 Locally: Extremely High VRAM and RAM Usage Reported — Full_Astronomer_5438 · 2026-08-03
- Cloudflare Launches Billable Usage API for Programmatic Cost Visibility — ritakozlov · 2026-08-03
- Cloudflare Workers & Containers Add Inbound TCP and gRPC Support — ritakozlov · 2026-08-03
- Cloudflare Launches @cloudflare/computer: A Dedicated Runtime Environment for Every Agent — threepointone · 2026-08-03
- Turso Database Overcomes SQLite Limits with Concurrent Writes — glcst · 2026-08-03