Tim Dettmers teases bitsandbytes2 with lazy compression for near-infinite KV cache
Tim_Dettmers · x · 2026-09-22
Tim Dettmers announced a slight delay to bitsandbytes2, now part of an upcoming open-source week releasing an entire ecosystem.
The headline feature is lazy compression: a technique requiring zero user configuration that always yields an optimal memory-quality-speed trade-off when deploying any model, even as the KV cache grows. Combined with another technique releasing the next day, it enables virtually infinite KV cache.
He credited students Eulrang Cho, Trang Nguyen, and Christine Park for their patience in holding back the releases.
Related event: bitsandbytes2 Slightly Delayed, Promising Near-Infinite KV Cache(2 posts)→
More from Infra
- Agent Substrate Calculator Estimates Session Resume Speed Under Load — pjausovec · 2026-11-12
- Dell expands AI Data Platform: agents have a data-context problem, not a model problem — DavidLinthicum · 2026-11-12
- SGLang Megakernel Pushes Muse Spark to 1,109 tok/s, 2.1x Over Baseline — hsu_byron · 2026-11-12
- Potion UI's local login system turns your PC into a multi-user AI server — cocktailpeanut · 2026-11-12
- AWS Finally Built Its Own AWS Emulator — super_linear · 2026-11-12
- Megakernel is multiprogramming for GPUs: 'GPUs are entering the CPU age' — pranjalssh · 2026-11-12