Moonshot’s Mooncake serving stack boosts long-context throughput by up to 525%
stochasticchasm · x · 2026-07-28
Mooncake is a KV cache–centric serving architecture for Kimi, Moonshot AI’s LLM service.
The paper describes a disaggregated design that separates prefill and decoding clusters and uses underutilized CPU, DRAM, and SSD resources in the GPU cluster as an external KV cache pool.
Main ideas:
- keep active decoding blocks in GPU KV cache
- write reusable idle prefixes back to CPU DRAM only after eviction
- prefetch them before reuse
- offload training states to NVMe after each training iteration to free DRAM
- add a prediction-based early rejection policy for overloaded rollout scenarios
Reported results:
- up to 525% throughput improvement in some simulated long-context scenarios
- 75% more requests handled under real workloads
The paper is especially focused on long-context serving and long-context RL rollouts.
More from Infra
- fmgo: call Apple's on-device Foundation Models from Go with no CGO and no Swift — Super_Run_8466 · 2026-09-23
- Huawei unveils Peerium architecture: nested BSP unifies million processors into one computer — Dr_Singularity · 2026-09-23
- Grok explains why DeepSeek picked DualPipe + ZeRO-1 over ZeRO-3 on 2048 H800s — TheZachMueller · 2026-09-23
- AI costs fall 47% per quarter, 4x faster than DNA sequencing: Epoch AI — daveholtz · 2026-09-23
- M5 Ultra LLM test: 4x faster prompt processing, but double the power draw — DigitalguyCH · 2026-09-23
- $500 of Dell OptiPlexes become a diskless netboot lab where AI agents can't brick the hardware — colinmcnamara · 2026-09-23