189 benchmarks later: how to pick KV quants for Qwen3 27B on 16GB VRAM
Fancy-Snow7 · reddit · 2026-10-15
A Redditor ran 189 benchmarks across 21 Qwen3 27B quantized variants and 9 KV quant settings that fit in 16GB VRAM, scoring each with KLD and perplexity.
Key findings
- q51 model + q51 KV is almost always better than mixed setups like q80/q40 or q40/q80, which are nearly identical to each other.
- Prefer raising the model quant and running q40/q40 over running a lower quant with f16 KV: max out the model quant first, then spend leftover VRAM on KV quant.
- Caveats: going q6→q8 has diminishing returns, so KV quant may matter more there; Gemma-style models requiring f16 KV are exempt. Bonsai/swift quants are finetunes and can't be fairly compared via KLD.
More from Infra
- Tokenmaxxing BERT: how skipping ASCII normalization made tokenization 50x faster — tomaarsen · 2026-10-15
- patio11 on multi-agent dev: clearly the future, and we'll buy so much inference — IgorCarron · 2026-10-15
- Hugging Face Hub makes model tensor inspection up to 20x faster, DeepSeek-V3.2 drops from 9.2s to 0.45s — victormustar · 2026-10-15
- RTX 5060 Ti 16GB surges up to 144% in Japan, hitting $1,292 as RTX 5080 tops $2,400 — chemist_slime · 2026-10-15
- Fish Audio Drama 3 lands in vLLM-Omni: 128 concurrent streams on one H200 — vllm_project · 2026-10-15
- 1,174 Macs form Darkbloom inference network, growing 4.5x in 60 days at half the cost — gajesh · 2026-10-15