KV cache eviction reframed as expectation estimation with importance weighting
guyvdb · x · 2026-09-03
Paper 'A Probabilistic Interpretation of KV Cache Eviction' (featured by alphaxiv):
- Most KV eviction methods use top-k heuristics to decide what to keep, which can heavily distort attention after tokens are removed.
- The paper reframes eviction as expectation estimation, using probabilistic sampling and importance weighting to correct for missing KV entries during decoding.
- This yields a principled bias-variance tradeoff that stays competitive with existing methods while being more robust.
Related event: Paper gives KV cache eviction a probabilistic foundation(3 posts)→
More from Infra
- TRL async GRPO adds LoRA sync via storage bucket and proxy, cutting 500-step training to 53 min — SergioPaniego · 2026-09-22
- GGUF models can now run directly in Hugging Face transformers with ggml Metal kernels — pcuenq · 2026-09-22
- Tech men joke about footwear after Allbirds pivots into a data center company — chrisalbon · 2026-09-22
- Modular's LLM Inference Handbook adds a GPU architecture primer for kernel work — blaizedsouza · 2026-09-22
- Quantized GLM MoE model (W4A16 AWQ) trends on Hugging Face — AikidoSec · 2026-09-22
- OpenRouter silently ignores effort params — benchmark via native API endpoints, not assumptions — PawelHuryn · 2026-09-22