llama.cpp PR Boosts Q2_0 CPU Decoding by 3x, 8B Hits 8.20 tok/s
BTA_Labs · reddit · 2026-08-07
A new llama.cpp PR (#26348) introduces an x86 VNNI implementation for the Q20 × Q80 dot product, achieving a massive 3.0x to 3.6x throughput improvement on x86 CPUs.
- Setup: Benchmarks were performed on an AMD EPYC 9645 (8 cores) with GPU and BLAS disabled, using Bonsai GGUF models.
- Performance Leap: 8B model decode speed jumps from 2.39 to 8.20 tok/s; the 1.7B model sees a 3.26x speedup.
- Consumer Hardware Potential: The reference implementation revealed that Intel 12th-14th Gen desktop CPUs (e.g., i5-13400) were silently missing the fast path due to fused off AVX-512. Fixing this yields a similar 3.2x boost.
- Caveats: The PR is currently open, the optimization is specific to Q20 quantization, and there are tiny numerical differences at the kernel level due to fused multiply-add behavior.
More from Infra
- 3x DGX Spark cluster runs GLM-5.3-Flash locally behind a proxy, 1.75M tokens/day at 119W — Teknium · 2026-10-16
- Optuna creator Akiba: when 2 hyperparameters cost $20M, what's left for BO/HPO — Stefania_druga · 2026-10-16
- Amazon Proposes Per-Request Validated Evidence Caching for Production RAG — _reachsumit · 2026-10-16
- LinkedIn's TransX report: sequence transduction for recsys, +6.02% CTR in A/B tests — _reachsumit · 2026-10-16
- llama.cpp PR boosts CPU inference speed with merged AVX2 tiled accumulators — jacek2023 · 2026-10-16
- Zero's next-gen IVM engine runs in a single DO, live-querying a 100GiB R2 database — threepointone · 2026-10-16