llama.cpp PR Boosts Q2_0 CPU Decoding by 3x, 8B Hits 8.20 tok/s
BTA_Labs · reddit · 2026-08-07
A new llama.cpp PR (#26348) introduces an x86 VNNI implementation for the Q20 × Q80 dot product, achieving a massive 3.0x to 3.6x throughput improvement on x86 CPUs.
- Setup: Benchmarks were performed on an AMD EPYC 9645 (8 cores) with GPU and BLAS disabled, using Bonsai GGUF models.
- Performance Leap: 8B model decode speed jumps from 2.39 to 8.20 tok/s; the 1.7B model sees a 3.26x speedup.
- Consumer Hardware Potential: The reference implementation revealed that Intel 12th-14th Gen desktop CPUs (e.g., i5-13400) were silently missing the fast path due to fused off AVX-512. Fixing this yields a similar 3.2x boost.
- Caveats: The PR is currently open, the optimization is specific to Q20 quantization, and there are tiny numerical differences at the kernel level due to fused multiply-add behavior.
More from Infra
- Cloudflare Unifies Workers AI and AI Gateway into a Single Control Plane — michellechen · 2026-08-07
- Running MiniMax on B200 GPU: 10s Video in Under 2 Minutes — Foreforks · 2026-08-07
- LiquidAI LFM2.5-2.6B Quantization Report: Runs on Raspberry Pi — crusaderky · 2026-08-07
- Turso Rewrites Postgres in Rust to Build the LLVM of Databases — JeremyCMorgan · 2026-08-07
- Cloudflare Unifies Workers AI and Gateway, Paving Way for Model-First Routing — Cloudflare Blog · 2026-08-07
- Rumor: OpenAI to Launch GPT-6 Next Week; Tesla Invests Billions in Terafab — Not Boring (Packy McCormick) · 2026-08-07