KV Cache Quantization Tested Across 35 Quants: 8-bit Is Essentially Free for Qwen3.8 27B
Hefty_Wolverine_553 · reddit · 2026-11-08
A systematic KLD study of Qwen3.8 27B quantization covering 35 weight quants across llama.cpp, ExLlamaV3 and vLLM with 16/8/4-bit KV caches, using a dataset with agentic tool-use traces and up to 64k contexts. Key findings: 8-bit KV cache halves cache size with essentially no KLD difference; 4-bit KV is measurable and not recommended, with impact growing 15-30% at 16-32k contexts; vLLM's official NVFP4/FP8 checkpoints underperform smaller GGUF/EXL3 quants mainly due to W4A4 activation quantization — rerunning via Marlin kernels cuts NVFP4 KL from 0.0256 to 0.0117.
More from Infra
- SpaceX plans 6 GW AI campus near Willcox, Arizona with dry cooling — aronchick · 2026-11-08
- François Fleuret asks: after a loss spike and recovery, is final loss worse, same, or better? — francoisfleuret · 2026-11-08
- ResearchBoost Agent speeds up real research workloads 2.51x on average across 12 tasks — YouJiacheng · 2026-11-08
- Dev builds Cray-2 emulator from scratch, runs a GPT-3-class LLM at 1 token per 4s — inductionheads · 2026-11-08
- Hyperbolic CEO on compute market-making: bridging 3-year contracts and 3-week needs — YiMaTweets · 2026-11-08
- Ben Bajarin mocks 'we don't need data centers' take: CSPs were 'useless' too — BenBajarin · 2026-11-08