llama.cpp PR brings ~1.3x Q2_0 prefill speedup on AVX-VNNI CPUs, big MoE gains too
pmttyji · reddit · 2026-11-16
llama.cpp PR #26348 adds an x86 AVX-VNNI dot-product kernel for Q20 quantization, delivering major CPU inference gains. On an AMD EPYC 9645 (8 cores, 16GB RAM), Bonsai models from 1.7B to 27B saw roughly 1.3x faster prefill (e.g. 8B up from 30.0 to 39.4 tok/s pp512), with tg128 gains of 1.09–1.22x.
The author expected the boost to be limited to ternary models, but it isn't: community reports show faster decode for IQ1S/IQ2XXS DeepSeek v4.1 as well. In a hybrid setup (RTX 4070 Laptop, --cpu-moe, 10 threads, MTP on) running a 66GB Qwen3.8-Flash-Next GSQ-RCO Q20 MoE, the CPU path improved from 3.1–4.1 t/s (generic scalar) to 14.0–18.7 t/s (AVX2), and 15.5–21.4 t/s with this PR's kernel.
Users running Q1/Q2 quants of large models are advised to re-benchmark, since GPU+CPU hybrid inference also benefits. A follow-up PR (#32307) adds 512-bit AVX-512 VNNI dot products for further speedups.
More from Infra
- Enterprise AI economics: cost the whole workflow, not just per-token pricing — kashifmanzoor · 2026-11-16
- Inference providers are not hosting Perplexity's decider model despite production demand — bo_wangbo · 2026-11-16
- Engineer job-hunting with Pony, a Go runtime that supervises multiple AI coding agents — sharpeye_wnl · 2026-11-16
- NYT: Jensen Huang's year-long secret about Megaspeed ties led to his downfall at Nvidia — GaryMarcus · 2026-11-16
- Bindu Reddy: Nvidia's 80% margin unsustainable as Neocloud financing props up its own demand — bindureddy · 2026-11-15
- Jensen Huang personally courting candidates in quiet Nvidia CEO succession search — WaPo — Polymarket · 2026-11-15