Salesforce research: Random eviction of reasoning tokens matches selective KV cache compression
Salesforce · hf · 2026-09-04
Salesforce's "Random Attention" paper challenges conventional KV cache compression: with the prompt preserved, randomly evicting reasoning tokens performs on par with carefully scored selective compression methods.
- Key insight: reasoning traces are "self-protecting" — critical information is naturally redundant across long chains of thought, so random drops cause little unrecoverable loss
- This removes the need for per-token importance scoring, cutting overhead and simplifying cache management for long-CoT inference
- Directly relevant to serving-stack optimization for long-reasoning models: a much simpler eviction strategy achieves near-parity
Related event: Study: Random KV Cache Eviction Beats Scoring Methods(2 posts)→
More from Infra
- PostgreSQL 19 slips to November as LLM-assisted code review floods committers with edge cases — sanjaykalra · 2026-10-21
- Decision Models' routing classifier hits AntSeed, cutting inference cost to 24% of always-Opus — airesearch12 · 2026-10-21
- Eon cuts engineer onboarding from 2 days to 2 minutes with an MCP Gateway on Bedrock AgentCore — AWS ML Blog · 2026-10-20
- a16z's Ben Horowitz says AI buildout isn't the 90s bubble — an investor pushes back — henloitsjoyce · 2026-10-20
- Micron and Nvidia drive ~36% of S&P 500's Q3 earnings growth — Beth_Kindig · 2026-10-20
- NVIDIA lands in Europe: Jensen Huang and German chancellor discuss industrial AI at GTC Berlin — nvidia · 2026-10-20