Block KV cache streaming PR bounds VRAM at long context in llama.cpp fork
giveen · reddit · 2026-09-06
Developer giveen submitted PR #357 to llama-cpp-turboquant implementing block KV cache streaming via a shared CUDA phase arena, bounding VRAM usage at long context. The author credits Raymond Huang's llama.cpp-adaptive-kv-streaming as the original work, ported it to turboX, extended model support beyond Qwen, and benchmarked thoroughly to confirm the gains.
More from Infra
- SurfSense: an open-source local NotebookLM with MCP agents and a prompt-caching lesson — FurtiveMirth · 2026-10-23
- Meta researcher: horrifying infra, not talent, is why OSS can't catch closed frontier — A_K_Nain · 2026-10-23
- Jane Street and NVIDIA speed up Mamba-2 with custom cuTile kernels — blelbach · 2026-10-23
- LatentLoop: asymmetric compute allocation cuts looped-MoE training cost by 19% — pmttyji · 2026-10-23
- Same AI request emits 8.5x more CO2 in Texas than in Sweden — jlundberg · 2026-10-23
- SK Hynix Pulls Plug on Solidigm US IPO as Chip Stocks Slide, Shares Down ~30% This Quarter — rohanpaul_ai · 2026-10-23