KV Cache Compression Makes 1M Context Manageable at Under 100GB Per User
khademinori · x · 2026-10-19
Responding to a discussion on context windows, AI practitioner slbai argues the industry has been stuck at 1M tokens for too long: modern KV cache compression makes 1M context fully manageable at datacenter scale, now under 100GB per user. He'd like to see at least a Sonnet-class model reach 2.5M tokens before year's end.
More from Infra
- DeepInfra's B200s sat idle at $1.49/hour even with promo videos — gharik · 2026-10-19
- SK Hynix on HBF: roughly half of HBM bandwidth but 14x the capacity — Beth_Kindig · 2026-10-19
- Elon Musk mocks what it's like trying to rent AI compute today — elonmusk · 2026-10-19
- New blog series part 1: a visual introduction to tensor core arithmetic — akbirthko · 2026-10-19
- Weekly recap: five vendors ship agent controls at five layers, desk-side AI draws most readers — shashib · 2026-10-19
- Hugging Face kicks off Open Source AI Week with ExecuTorch edge-AI hackathon — Thom_Wolf · 2026-10-19