Random KV Cache eviction rivals top baselines and boosts vLLM throughput 32-43%
机器之心 · wechat · 2026-09-19
A new paper from Salesforce AI Research and UIUC introduces RandomAttention: fully protect the input prompt, then keep the remaining reasoning-trace KV entries via per-head random sampling, with no content-based importance scoring.
The results are counterintuitive: across Qwen3-4B/14B/32B and Phi-4-reasoning on six math, science, and coding tasks at 4× cache compression, it matches the strongest baselines, significantly winning 31 of 60 comparisons and losing only 1. Integrated into vLLM for 32k-token serving, skipping the scoring pass lifts throughput 32–43% over TriAttention.
The paper then explains why: controlled experiments show much of prior methods' gains came from accidentally protecting the prompt (unified prompt protection shrinks SnapKV/VaSE gaps to under 2.2 points), and reasoning traces carry two layers of redundancy — textual restatement and cross-head copies. A planted-fact probe shows retrieval hits 99% when a fact survives in 8 heads vs 3% in one. Yet for facts stated once and never restated (a passcode probe), RandomAttention scores zero, showing selection signals still matter for needle retrieval.
More from Infra
- DiffusionGemma-based DJev runs near-real-time vision detection on a phone — PMinervini · 2026-09-19
- 35B ternary LLM runs on iPhone in ~4GB RAM: Millie beats Bonsai 27B, 60% faster — MannyKayy · 2026-09-19
- Dev hacks llama.cpp for NVFP4 KV cache, runs Qwen3 27B at 262k context across two GPUs — comperr · 2026-09-19
- PyTorch Conference to showcase CRCR, letting downstream repos plug into upstream CI — PyTorch · 2026-09-19
- First Cafe Compute Nairobi meetup demos Cerebras API, calls for an /explain interpretability endpoint — paw_lean · 2026-09-19
- Cerebras launches Money Agent, a Qwen 3 27B-powered personal finance assistant — Alibaba_Qwen · 2026-09-19