Kimi K3 report details dual cache inference and snapshot-based sandbox infra
nrehiew_ · x · 2026-07-29
The thread describes Kimi K3’s efficient inference stack, focusing on how it handles two cache types at once: KDA state and MLA KV cache.
- Because KDA uses a recurrent state shape, block-level hashing does not work well, so the system uses different hashing granularities for MLA and KDA.
- A figure shows fine-grained prefix caching within a 6144-token physical block, where persisted KDA checkpoints are sparse and tied to conversation boundaries.
- The poster also notes K3’s sandbox infrastructure, highlighting secure execution and efficient pause/resume from snapshots.
- The post further mentions offloading external KV cache to CPU as prefix cache and training states to NVMe, though some details are uncertain in the thread.
More from Infra
- Meta's open Muse Spark needs 493GB of memory — far beyond any consumer laptop — TejasKumar_ · 2026-10-16
- CoreWeave plunges 20% as credit markets price >60% default odds; is this dot-com all over again? — kimmonismus · 2026-10-16
- MiniCPM-V 4.7 35B-A3B hits 17+ tokens/s on iPad with ~3GB memory — techNmak · 2026-10-16
- China rolls out 'token factories' and a 2026-28 token economy action plan — nordicinst · 2026-10-16
- Vercel Sandbox Drives go GA: attach TBs of data in sub-second, fork instantly — cramforce · 2026-10-16
- NVIDIA launches RTX Spark for local AI on Windows PCs — ostrisai · 2026-10-16