KVarN Reduces VRAM for 27B Long Context
logickkk1 · hn · 2026-07-15
This post shares benchmark results from porting the **KVarN** structured KV cache quantization to the **Bonsai** runtime, aiming to reduce VRAM usage and boost generation speed for long contexts. ### Core Findings - On **Bonsai-27B Q1_0**, **120K context**, **RX 9070 XT**: - `q8_0` cache: **43.4 tok/s**, total VRAM ~**13.1GB** - `kvarn4` cache: **73.0 tok/s**, total VRAM ~**9.8GB** - Compared to the original method, the author reports a **+68% speed boost** and ~**3.3GB VRAM saved**. ### Method - KVarN is based on concepts from **KIVI** and **QuIP#**. - It applies quantization after **Walsh-Hadamard transform** and **Sinkhorn variance equalization** to mitigate quality loss from low-bit KV caches. ### Practical Observations - The author believes this approach is more valuable for scenarios requiring **long context retention**. - They compared it to **TurboQuant**, noting that while the latter is faster, it carries a higher quality risk. - They also mention this modification **100% breaks DFlash**, but it doesn't impact their use case. ### Reproduction Info - The author ported the solution to the Bonsai runtime, modifying about **23 files / 1300 lines**. - Recommended for trying dense model long-context inference in Bonsai or similar setups: `--cache-type-k kvarn4 --cache-type-v kvarn4`.
More from Infra
- Larry Fink says China is ahead in the AI energy race, citing 100 GW nuclear buildout — rohanpaul_ai · 2026-07-21
- Local AI may pay back in 6–7 years and cut long-term costs by 30–40% — DavidLinthicum · 2026-07-21
- TSMC reportedly plans up to 10% chipmaking price hikes in 2027 — kimmonismus · 2026-07-21
- More open models and llama.cpp updates are coming, says Merve Noyan — mervenoyann · 2026-07-21
- Why adding a second LLM provider breaks more than the API surface — Ok_Extension6373 · 2026-07-21
- UK AI datacentres face backlash over heat, noise and land use — nordicinst · 2026-07-21