OpenLake says external KV cache offload cuts long-context inference cost by 48%
arnav__1 · hn · 2026-07-26
OpenLake, an open source storage engine for offloading LLM KV caches to shared RAM and NVMe, says it can cut long-context inference cost nearly in half.
- A 256K-token Gemma 4 31B conversation can generate about 43GB of KV state, which can exceed half of an 80GB H100.
- The system targets a cluster-level problem: cached prefixes are often unavailable when requests land on a different GPU host, forcing repeated work.
- To reduce network traffic, OpenLake uses deferred materialization: a custom CUDA kernel that losslessly compresses KV blocks before they leave GPU memory and decompresses them on retrieval.
- Reported results:
- 1.72× lossless KV compression
- 600GB/s decompression throughput on H100
- 80GB/s effective KV throughput over a 50GB/s physical link
- TTFT reduced from 44s to 0.6s at 128K context
- GPU time cut from 1,169s to 606s, saving 48.2% of GPU cost
- It is written in Rust, uses iouring, and ships connectors for vLLM and SGLang.
More from coding & agent
- Specialized APIs are the real edge for profitable AI agents, not vanilla models — eptwts · 2026-07-26
- Local AI agent demo syncs sessions, remote phone control, and working tool calls — SIGKITTEN · 2026-07-26
- A Chrome extension got pricier, so he rebuilt it in Claude in 15 minutes — Delahuntagram · 2026-07-26
- ComfyUI workflow uses LTX 2.3 and IC-LoRA to rebuild video from pose and one frame — waterarttrkgl · 2026-07-26
- Codex used computer control to price old items, draft listings, and post them — kevinkern · 2026-07-26
- Codex app gets a rave review as its design and experience finally feel polished — lucasmeijer · 2026-07-26