bitsandbytes2 slightly delayed: zero-config lazy compression plus dynamic expert swaps for near-infinite KV cache
Tim_Dettmers · x · 2026-09-22
bitsandbytes2 was set to release this week but is slightly delayed. It offers "lazy compression" — no user configuration, always a perfect memory-quality-speed trade-off, even as the KV cache grows. Combined with another technique releasing tomorrow, it enables virtually infinite KV cache. bnb2 targets small companies and individuals deploying large open-weight models on limited-memory hardware; private beta signups are open.
Related event: bitsandbytes2 Slightly Delayed, Promising Near-Infinite KV Cache(2 posts)→
More from Infra
- H Company trains computer-use agents on SkyPilot: thousands of sub-second sandboxes — skypilot_org · 2026-09-23
- MiniMax H3 video gen runs locally on M5 Ultra: 768p in ~2m22s with optimizations — bakawolf123 · 2026-09-22
- Cisco: Agentic AI to Drive 9X Enterprise Traffic Growth by 2035 vs 2.5X Without — Beth_Kindig · 2026-09-22
- Cloudflare ships Vary support in Cache Rules to tame HTTP's 'ugliest' header — threepointone · 2026-09-22
- Fluidstack breaks ground on $4B Texas data center campus planned for up to 1.5 GW — MxMnr · 2026-09-22
- Right-size GenAI endpoints: SageMaker concurrency sweeps with Nemotron-3 Nano 30B — AWS ML Blog · 2026-09-22