Future Trends in AI Inference Memory and Caching
AccBalanced · x · 2026-07-11
In a podcast episode, Vik and Val Bercovici discussed the evolution of AI inference memory and caching. Although the cost of KV cache continues to drop significantly, the resulting Jevons paradox has caused a massive surge in usage (for every 100x cost reduction, usage increases by about 10000x), keeping overall compute demand on the rise.
Technical details and insights include:
- Network and Motherboard: NVLink beats traditional PCIe (which has only 32 lanes) with 128 lanes.
- Storage Solutions: Networked NAND provided by WEKA offers storage speeds even faster than network DRAM. It pools NAND via RDMA/NVLink, avoiding the limitations of CXL's need for a dedicated bus.
- DeepSeek Costs: DeepSeek's cache read costs are extremely low (about 87x cheaper, but limited to the China region).
- Industry Predictions: SaaS giants and emerging cloud service providers will inevitably merge in the future.
Related event: AI Inference Faces KV Cache and Bandwidth Bottlenecks(3 posts)→
More from Infra
- oMLX 0.5.2 adds Mac menu-bar stats, low-bit decode kernels, and faster downloads — awnihannun · 2026-07-22
- Strangeworks launches Aura to turn enterprise ops into production optimization systems — whurley · 2026-07-22
- Graph workload 854.graph500 enters SPEC CPU 2026 as a new CPU benchmark — Prof_DavidBader · 2026-07-22
- Hybrid and local inference are emerging as a response to AI energy and token costs — dmitry140 · 2026-07-22
- NVIDIA details Vera CPU with 2x performance claims and a 22,000-core rack — ryanshrout · 2026-07-22
- NVIDIA says Vera Rubin NVL72 delivers 10x more tokens per megawatt than Blackwell — nvidia · 2026-07-22