KV Cache Scoring Is Useless: Random Eviction Matches Top Evictors at 32-43% Higher Throughput
burny_tech · x · 2026-09-04
A new paper, Random Attention, shows the dominant KV cache eviction paradigm—scoring cached tokens to decide what to keep—contributes almost nothing.
- Method: Keep the prompt's cache and evict reasoning tokens uniformly at random within each attention head, with no scoring at all
- Results: Matches the strongest prior evictor across 4 models and 6 reasoning tasks while delivering 32-43% higher throughput in vLLM deployment
- Why it works: The prompt is the fragile, critical part of the cache; reasoning traces protect themselves via textual redundancy and per-head copies, so random draws retain enough copies without any scoring
- Code and paper are publicly available
Related event: Study: Random KV Cache Eviction Beats Scoring Methods(2 posts)→
More from Infra
- Turso unveils MVCC implementation to lift SQLite's single-writer bottleneck — glcst · 2026-10-21
- Railway pauses all new builds amid major outage, users ask if vibe coding is to blame — AaronBergman18 · 2026-10-21
- a16z: Superhuman AI hardware design is the easy part — manufacturing is the hard one — a16z · 2026-10-21
- Sizing a 1 GW datacenter: ~522,000 Blackwell GPUs fit inside — bookwormengr · 2026-10-21
- DeepSeek's production density reportedly hits ~$38 per GPU-hour at saturation — bookwormengr · 2026-10-21
- PyTorch details KTIR and Dataflow Scheduler behind IBM's Spyre accelerator backend — PyTorch · 2026-10-21