Resin adds tiered storage for KV cache, letting CPU memory extend GPU KV capacity
andersonbcdefg · x · 2026-10-31
The open-source inference engine Resin now supports tiered storage for KV cache: KV and recurrent state evicted from GPU memory can spill to host CPU memory or other tiers, with async fetches for reuse.
The Engine handles prefix matching and device-page lifetimes while storage providers keep immutable checkpoints outside the device cache pool. A bounded CPU DRAM provider (MemoryCacheStorage with maxbytes, using pinned host memory) ships built-in, and since it is "just code" behind a lookup/store/load interface, custom providers for SSDs, Mooncake, etc. are also possible.
For long-context and multi-turn workloads, this effectively extends KV capacity with cheap memory and boosts prefix cache hit rates.
More from Infra
- Nvidia weighs cosigning loan for Nscale's West Virginia AI data center — TechCrunch AI · 2026-10-31
- Northflank dunked on Railway's breach, then got hacked the same way 9 days later — aniketmaurya · 2026-10-31
- Dev says Neon is already chasing millisecond infra branching, wants it built into alchemy — samgoodwin89 · 2026-10-31
- Big AI labs all push compute efficiency: Trainium power savings, small models, cache compression — WillfullyWorkable · 2026-10-31
- Running 12 agents on one RTX 3090 lifts total throughput from 30 to 110 tok/s — Mchanger · 2026-10-31
- Analyst turns ultra-bull on Avicena after 6 hours of technical diligence — BenBajarin · 2026-10-31