Model quant vs KV quant: Which tradeoff is better?
Elorun · reddit · 2026-08-20
A user is weighing two quantization setups for Qwen 3.8 27B on llama.cpp while maintaining a context length above 150k: Q4KM weights with Q8 KV cache, or Q4KXL weights with Q4 KV cache. The question is whether it's better to use larger weight quantization with lower KV cache precision, or smaller weight quantization with higher KV cache precision.
More from Infra
- Large data centers employ only ~50-150 permanent on-site staff, Microsoft says — brezina · 2026-10-16
- Prediction: Opus 5.5-level models on laptops by 2027, backed by Qwen 27B data — cephaloform · 2026-10-16
- Oracle down 12% in a week, nearly 50% off June high as OpenAI exposure spooks market — alvelda · 2026-10-16
- US data center moratoriums top 400 — an AI bull explains why they're actually good for AI — kevinsxu · 2026-10-16
- Microsoft Research unveils Magnetar, the fastest zkVM at 10-39x speedups and post-quantum security — jedisct1 · 2026-10-16
- Bonsai 2 2B hits 120 tok/s on iPhone 18 Pro, outpaces Qwen 3.5 2B at half the memory — cephaloform · 2026-10-16