Gandalf paper: KV cache offloading wears out SSDs in months; attention-aware admission extends life 7.3x
ThePrimeClock · reddit · 2026-10-26
A new paper, Gandalf (arXiv:2610.30791), tackles SSD wear caused by KV cache offloading in local LLM serving. In production workloads like multi-round chats and agents, total KV cache writes far exceed SSD endurance, shrinking flash lifetime from years to months—increasingly relevant as new local inference models bring SSDs into the hardware stack.
- The obvious fix—gating KV writes by historical access patterns—hurts cache hit rates, since caches from different attention types can share history but diverge in future access patterns.
- Key insight: admission decisions should combine access patterns with application-level info, namely the attention type of each KV cache.
- Gandalf is the first SSD lifetime-aware KV cache offloading system, prioritizing full-attention KV caches for SSD admission.
- Evaluation: SSD lifetime extends by 7.3x on average with negligible hit-rate impact across workload traces and hardware capacities; the design extends to other flash media like High Bandwidth Flash (HBF).
More from Infra
- Investor questions like "Are GPUs waterproof" mocked as neocloud hype peaks — arthurcolle · 2026-10-26
- DeepSeek V4.1 Flash earns $38/GPU-hr vs Kimi's $8.34 — a 4.56x gap reshaping inference — bookwormengr · 2026-10-26
- Matthew Berman: Image, video and voice gen are now good enough to run locally — MatthewBerman · 2026-10-26
- Reddit asks: Why can't AI coding agents run locally like Stable Diffusion? — Kaporalhart · 2026-10-26
- Vjeux joins Muse Open Kernel Challenge; colinlikescode leads 4 of 5 problems — Vjeux · 2026-10-26
- Gary Marcus publicly wagers on whether CoreWeave stock drops below $40 this week — GaryMarcus · 2026-10-26