Cloudflare's LLM Serving Math: FP8 and INT4 Quantization Double Concurrency
arpit_bhayani · x · 2026-08-05
The author breaks down Cloudflare's blog on serving Kimi K2.6 and GLM 5.2 at scale, noting that KV cache typically exhausts GPU memory before model weights do.
Core Optimization Strategies:
- Kimi K2.6: Switching KV cache from BF16 to FP8 halves its size, boosting max in-memory context from 686k to 1.37M tokens. Although FP8 is slightly slower per token (5-9%) due to conversion overhead, halving the footprint increases max concurrency from 32 to 64, yielding a 41% higher peak throughput.
- GLM 5.2: Compressing model weights from FP8 to INT4 reduces per-GPU memory from 88GB to 52GB.
Benchmarks confirm negligible quality loss: FP8 KV cache scores within a point of BF16, and INT4 weights stay within 0.8 points of FP8.
More from Infra
- Local H3 video gen: 15s clip in ~5 min at zero cost vs ~$0.60 per generation on platforms — AIandDesign · 2026-09-22
- MotherDuck makes text classification 50x faster at ~1% of LLM cost — josh_wills · 2026-09-22
- Google serves 3.2 quadrillion AI tokens a month — ~12GW, and energy could run out in 3 years — victor_explore · 2026-09-22
- M5 Ultra 256GB early test: Mimo2.6-Flash hits 49 tok/s at 32K context — bakawolf123 · 2026-09-22
- TRL async GRPO adds LoRA sync via storage bucket and proxy, cutting 500-step training to 53 min — SergioPaniego · 2026-09-22
- GGUF models can now run directly in Hugging Face transformers with ggml Metal kernels — pcuenq · 2026-09-22