RandKV ships as pip-installable random KV-cache eviction for Transformers, reports honest negative perf results
atease01 · reddit · 2026-09-07
A developer released RandKV, a pip-installable implementation of the Random Attention KV-cache eviction policy for Transformers, built after struggling to reproduce the paper without its full research harness.
- Implementation: protects the full prompt, retains a recent-token buffer, and independently samples older generated tokens per KV head; absolute sequence positions stay separate from compacted physical cache length, with deterministic seeds derived from request, eviction round, layer, and head.
- Honest negative result: on Qwen3-0.6B with an Apple M4, the current Python/PyTorch compaction path measured 35.48 tok/s vs 38.53 tok/s dense (0.921x) — evidence of adapter overhead and compatibility, not validation of the paper's reported vLLM throughput gains.
- Scope: Transformers 5.16 only, batch size one, full-attention decoders, no beam search, no vLLM backend yet.
Next milestone is a CUDA/vLLM backend with matched quality and serving benchmarks; feedback on the cache interface and benchmark design is welcome.
More from Infra
- 928-Star Wiki Details Running Qwen3.5-397B and Kimi-K2.5 on NVLink-Free PCIe RTX 6000 — TheZachMueller · 2026-09-07
- Prediction: compute is moving from rack-scale to datacenter-scale as bottlenecks shift outward — AccBalanced · 2026-09-07
- AI meme: turning off the tap while brushing teeth to "conserve water for the datacenter buildout" — EigenGender · 2026-09-07
- AMD MI355X beats NVIDIA B300 on tokens-per-dollar in AgentX — AccBalanced · 2026-09-07
- Offline village AI: $5,000 budget to build a local LLM machine for basic Q&A, seeking GPU advice — Potential_Low_1183 · 2026-09-07
- Custom llama.cpp Branch Adds Expert Expansion for MoE Models — Specific-Tax-6700 · 2026-09-07