Tim Dettmers teases bitsandbytes2 with lazy compression for near-infinite KV cache
Tim_Dettmers · x · 2026-09-22
Tim Dettmers announced a slight delay to bitsandbytes2, now part of an upcoming open-source week releasing an entire ecosystem.
The headline feature is lazy compression: a technique requiring zero user configuration that always yields an optimal memory-quality-speed trade-off when deploying any model, even as the KV cache grows. Combined with another technique releasing the next day, it enables virtually infinite KV cache.
He credited students Eulrang Cho, Trang Nguyen, and Christine Park for their patience in holding back the releases.
Related event: bitsandbytes2 Slightly Delayed, Promising Near-Infinite KV Cache(2 posts)→
More from Infra
- Ben Bajarin launches expert interview series on powering AI datacenters with gas — BenBajarin · 2026-09-23
- umbrelOS 2.0: your data belongs at home, now paired with local LLMs and agents — JosephJacks_ · 2026-09-23
- umbrelOS 2.0 launches, turning a small home computer into your personal cloud — JosephJacks_ · 2026-09-23
- fal's H3 Max generates 5 seconds of frontier-quality video in 3 seconds via full-stack optimization — gorkem · 2026-09-23
- Presenting Undersea Internet Cables at the United Nations — EricTopol · 2026-09-23
- NVIDIA's AI Factory Insider Ep 5: How CUDA Powers Specialized AI Across Industries — NVIDIA Developer · 2026-09-23