llama.cpp Introduces Dynamic VBR KV Cache
old-mike · reddit · 2026-07-14
This post introduces the **VBR (Variable Bit Rate) KV cache**, a new feature in spiritbuun's llama.cpp fork: - Instead of a fixed KV quantization level for the entire context, it sets a floor, and the cache dynamically steps down quantization levels as the context grows. - The `-ct vbr` command automatically calculates the available KV budget based on remaining VRAM and dynamically reduces precision during the decode process. - Tested on an RTX 3060 12GB compared to fixed quantization: - Decode TPS is only ~4% slower - TTFT is roughly halved - Lower VRAM usage - Automatically adapts to a context of ~216k tokens - The post also mentions a fused kernel bug, for which the author has already reverted the relevant commit.
More from Infra
- AI performance is increasingly limited by materials science, not just compute — nordicinst · 2026-07-21
- A GLM-5.2 inference debate asks how 750B parameters can exceed 1 token per second — francoisfleuret · 2026-07-21
- Why vector databases slow AI agents down after constant writes — PrajwalTomar_ · 2026-07-21
- Larry Fink says China is ahead in the AI energy race, citing 100 GW nuclear buildout — rohanpaul_ai · 2026-07-21
- Local AI may pay back in 6–7 years and cut long-term costs by 30–40% — DavidLinthicum · 2026-07-21
- TSMC reportedly plans up to 10% chipmaking price hikes in 2027 — kimmonismus · 2026-07-21