Cloudflare's LLM Serving Math: FP8 and INT4 Quantization Double Concurrency
arpit_bhayani · x · 2026-08-05
The author breaks down Cloudflare's blog on serving Kimi K2.6 and GLM 5.2 at scale, noting that KV cache typically exhausts GPU memory before model weights do.
Core Optimization Strategies:
- Kimi K2.6: Switching KV cache from BF16 to FP8 halves its size, boosting max in-memory context from 686k to 1.37M tokens. Although FP8 is slightly slower per token (5-9%) due to conversion overhead, halving the footprint increases max concurrency from 32 to 64, yielding a 41% higher peak throughput.
- GLM 5.2: Compressing model weights from FP8 to INT4 reduces per-GPU memory from 88GB to 52GB.
Benchmarks confirm negligible quality loss: FP8 KV cache scores within a point of BF16, and INT4 weights stay within 0.8 points of FP8.
More from Infra
- Anthropic Officially Confirms In-House Custom AI Chip Team for Claude — ns123abc · 2026-08-05
- Proxmox VE Launches Official ARM64 Support, Fully Compatible with NVIDIA Grace Hopper — jedisct1 · 2026-08-05
- Benchmarking MiniMax H3: Doubling Video Length Nearly Triples Render Time — jtreminio · 2026-08-05
- TensorSharp MoE Offload Slashes VRAM Use, Outperforms llama.cpp by up to 8x — fuzhongkai · 2026-08-05
- Oxford Economist: AI Arms Race Drives Massive Data Center Rollout — carlbfrey · 2026-08-05
- Cerebras Founder Breaks Down AI Chip Supply Chain and Inference Migration — mattturck · 2026-08-05