KV cache, not parameter count, may be the real bottleneck for long-context local models
jonejy · reddit · 2026-09-03
The author argues local-LLM discussions overfocus on parameter count while KV cache is the real memory bottleneck at long context: key/value states grow per token, so a model that fits in VRAM can break at 100k–200k context. GQA/MQA and KV cache quantization help but don't change linear growth. Future optimization will target memory movement and persistent state — designing models around 'how much they need to remember' rather than parameter count.
More from Infra
- One int8 op makes exact rescoring nearly free in PLAID-style late-interaction search — antoine_chaffin · 2026-09-03
- RX 6800 XT vs RX 9060 XT for ComfyUI: is the $105 premium worth it for LTX and Wan? — Nice-Regret-9207 · 2026-09-03
- Developers warn a new Cloudflare feature could hurt your SEO — turn it off — gaganghotra_ · 2026-09-03
- If Amazon Trainium is any good, why weren't they at Hot Chips? — firstadopter · 2026-09-03
- Dragonfly rethinks Redis with sharded multi-threading to scale across modern multi-core servers — techNmak · 2026-09-03
- Zeiss exec: China is about 15 years behind in cutting-edge chipmaking tools — broodsugar · 2026-09-03