Random KV Cache eviction rivals top baselines and boosts vLLM throughput 32-43%
机器之心 · wechat · 2026-09-19
A new paper from Salesforce AI Research and UIUC introduces RandomAttention: fully protect the input prompt, then keep the remaining reasoning-trace KV entries via per-head random sampling, with no content-based importance scoring.
The results are counterintuitive: across Qwen3-4B/14B/32B and Phi-4-reasoning on six math, science, and coding tasks at 4× cache compression, it matches the strongest baselines, significantly winning 31 of 60 comparisons and losing only 1. Integrated into vLLM for 32k-token serving, skipping the scoring pass lifts throughput 32–43% over TriAttention.
The paper then explains why: controlled experiments show much of prior methods' gains came from accidentally protecting the prompt (unified prompt protection shrinks SnapKV/VaSE gaps to under 2.2 points), and reasoning traces carry two layers of redundancy — textual restatement and cross-head copies. A planted-fact probe shows retrieval hits 99% when a fact survives in 8 heads vs 3% in one. Yet for facts stated once and never restated (a passcode probe), RandomAttention scores zero, showing selection signals still matter for needle retrieval.
More from Infra
- Malaysia poised to become a compute superpower with both US and Chinese chips by next year — basedjensen · 2026-11-04
- ByteDance details USDB storage system: 85.8% lower write latency for recommender sequences — _reachsumit · 2026-11-04
- Oracle cancels $7B chip leasing deal with Tencent, FT reports — SumitGup · 2026-11-04
- Inference engine shrunk to 1023 bytes, still runs SmolLM2-135M — cneuralnetwork · 2026-11-04
- Musk says space data centers are 'looking pretty appealing right now' — elonmusk · 2026-11-04
- Data centers are turning to old jet engines as the wait for power grows — IndependentCoast3974 · 2026-11-04