A proposal to make KV cache portable across machines, data centers, and the WAN
knowrohit07 · x · 2026-07-29
The post argues that portable inference state should work across machines, accelerators, data centers, and eventually the WAN. It says keeping all KV cache in accelerator memory is usually impractical, because moving raw tensors to local storage, another node, or S3 can erase the compute savings from reuse.
The proposed framing is to treat KV cache as structured temporal data rather than an opaque byte array. Because neighboring tokens produce related state and model layers/attention heads create stable structure, a video-codec-style mapping can preserve those relationships and remove redundancy across both space and time.
The design is described around a few principles:
- preserve token order as video time
- keep attention structure intact during encoding
- exploit redundancy across tokens, layers, and heads
- make the state portable instead of tied to one accelerator
Related event: Cradle Codec: GPU-Native KV Cache Compression for Cross-Node Migration(3 posts)→
More from Infra
- A beginner-friendly video explains five GPU optimization methods for LLMs — 2C_ornot2C · 2026-07-29
- Jon Durbin says he pre-trained a 20B MoE for under $10 an hour — const_reborn · 2026-07-29
- Local LLM Setup: Is a Modded 3080 20GB Worth It Next to a 3090? — YourNightmar31 · 2026-07-29
- DIY local AI server uses retired NVIDIA cards for about $165 total — blelbach · 2026-07-29
- Cheap local intelligence could shift AI workloads away from the cloud — PeterDiamandis · 2026-07-29
- Bull case says AMD profit could 10x as AI spend and inference demand scale — AccBalanced · 2026-07-29