FlexBudget poster at PyTorchCon: fair KV cache eviction comparisons by pinning kernel and budget
PyTorch · x · 2026-10-14
PyTorch previewed a PyTorchCon North America poster by AMD's Huanran Wang: FlexBudget: Interchangeable KV Importance Signals under One Compiled FlexAttention Budget.
- Problem: long-context inference requires evicting KV cache entries, but each method (keep recent tokens, keep high-attention tokens, etc.) ships welded to its own kernel — you can't tell if a smarter rule or just faster code won.
- Approach: hold the kernel and memory budget fixed, swap only the importance rule — same code, same allowance, honest comparison.
- Details: training-free, built entirely on core PyTorch (FlexAttention). Event runs Oct 20–21 in San Jose.
More from Infra
- Cooler Master launches 300 kW liquid cooling CDU, up to 450 kW, for GB200/GB300 racks — BenBajarin · 2026-10-14
- MSAR brings multi-head depth attention to residuals, beating AttnRes and mHC with 1.58x iso-loss speedup — SonglinYang4 · 2026-10-14
- Cloudflare's pipeline handles 2B events/sec, five years of optimization deferred 2.5 years of hardware growth — irvinebroque · 2026-10-14
- South Korea to deploy 74 portable Starlink systems for forestry by end of 2026 — XFreeze · 2026-10-14
- ThunderSyncRL speeds agentic RL up to 1.9x with zero policy staleness, Stanford paper claims — heghbalz · 2026-10-14
- Agent task costs fall ~90% every 6 months, countering Bloomberg's data center cost alarm — sudoraohacker · 2026-10-14