Impact of KV Quantization on Qwen3.6-27B Tested
BitGreen1270 · reddit · 2026-07-08
The author systematically tested the impact of different KV quantization combinations on KL Divergence (KLD) for Qwen3.6-27B (bartowski quantization) using an RTX 5090. Key findings: Q8 generally outperforms Q6 and Q5; performance for Q8 and Q6 degrades sharply if the value uses q40; and Q5 is more tolerant of value quantization. The final recommendation is to use the highest quantization your VRAM allows and stick to the (q80, q80) KV quantization configuration, which results in almost no performance loss.
More from Infra
- CoreWeave 5-year CDS spikes to ~1,100bps as market fears AI bubble starts as credit event — kimmonismus · 2026-10-26
- NVIDIA Plans ~$2,000 Per GPU Per Year Software Fee for Running AI Agents — LinusEkenstam · 2026-10-26
- 12 DevOps concepts to learn in order, from Linux to SRE — _jaydeepkarale · 2026-10-26
- Killed our Vector DB for SQLite BM25 and wrote a 3.6ns Rust sentinel: lessons from multi-agent loops — OpenComplaint524 · 2026-10-26
- Gemma 4 26B-A4B runs offline on a 16 GB Mac: 33/40 on TypeScript tasks at 31 tok/s — Mission-Sir-1614 · 2026-10-26
- How thousands of GPUs train one AI: a breakdown of 5D parallelism — jbhuang0604 · 2026-10-26