Inside a Trillion-Token/Day Factory: Mooncake Turns KVCache into a Cluster-Shared Pool
量子位 · wechat · 2026-09-11
QuJing/KVCache.AI details the inference architecture behind a trillion-token-per-day production system, where per-machine token efficiency rose 3x and total capacity 30x since Spring 2026 — powered by pooling KVCache across the cluster with MooncakeStore.
- Agentic workloads hammer long-context reuse; a single cache miss can force re-prefill of hundreds of thousands of tokens
- Architecture: Prefill-Decode separation, SGLang HiCache on Prefill only, MooncakeStore as an independent service pooling host DRAM, RDMA over 800Gbps NICs, engine/cache decoupled
- SMG-based cluster scheduling balances cache locality vs. hotspots; RBG orchestrates roles on Kubernetes
- Cache hit rate holds above 90%; average batched KVCache reads under 50ms via async prefetch overlapping compute
- War stories: LRU eviction's shard write-lock contention fixed by object-granular locks and moving slow ops out of critical sections; node scale-up/down accelerated several-fold via memory/RDMA registration optimizations and huge pages
More from Infra
- SemiAnalysis: OpenAI reportedly negotiating ~1GW TPU commitment while seeking sites — dylan522p · 2026-10-28
- I tested 11 video upscaling methods for AI video: local, Topaz and cloud compared — lopas99 · 2026-10-28
- Thermo-backed CTO says AI is in its dial-up phase, touts disaggregated compute for an ultrafast 'broadband moment' — beffjezos · 2026-10-28
- DeepSeek-V4.1-Flash Q2 runs at 41 tok/s on one RTX 3090, 90% token match vs API — musicurgy · 2026-10-28
- USearch v3 RC ships after 1,000 days, its libraries now embedded in 550K-star projects — viglovikov · 2026-10-28
- SK Hynix says memory demand will outpace supply through 2030 — Chemical_Hawk_6307 · 2026-10-28