A proposal to make KV cache portable across machines, data centers, and the WAN
knowrohit07 · x · 2026-07-29
The post argues that portable inference state should work across machines, accelerators, data centers, and eventually the WAN. It says keeping all KV cache in accelerator memory is usually impractical, because moving raw tensors to local storage, another node, or S3 can erase the compute savings from reuse.
The proposed framing is to treat KV cache as structured temporal data rather than an opaque byte array. Because neighboring tokens produce related state and model layers/attention heads create stable structure, a video-codec-style mapping can preserve those relationships and remove redundancy across both space and time.
The design is described around a few principles:
- preserve token order as video time
- keep attention structure intact during encoding
- exploit redundancy across tokens, layers, and heads
- make the state portable instead of tied to one accelerator
Related event: Cradle Codec: GPU-native KV Cache Compression for Ethernet Transfer(4 posts)→
More from Infra
- Neocloud Stocks Slashed Again: CRWV -10%, NBIS -9%, IREN -8% in One Day — JOBhakdi · 2026-10-16
- Neocloud stocks sink on Gartner's "80% will fail" report; analyst calls it garbage — McDonaghMatthew · 2026-10-16
- JapanFold Claims One Tenstorrent Server Beats an Nvidia DGX B200 on Bio Models at a Third of the Price — DavidBennett__ · 2026-10-16
- vLLM ships Day-0 RL support for Beam in SkyRL, +62% LoRA decode throughput — vllm_project · 2026-10-16
- DFlash2 speeds Beam decode up to 3.03× across benchmarks, vLLM reports — vllm_project · 2026-10-16
- Inside Beam: 501B params, only 23B active per token via 8-of-256 expert routing — vllm_project · 2026-10-16