llama.cpp Adds Q8_0 Support
pmttyji · reddit · 2026-07-15
ggml-zendnn added Q80 quantization support to llama.cpp and compared the throughput performance of GGML CPU versus ZenDNN across multiple models.
Key Takeaways
- During the prompt processing phase, ZenDNN shows significant improvements, especially with longer prompts.
- The decode phase (tg128) is basically on par with ggml-cpu.
- Tested models include Llama-3.1-8B-Instruct, Mixtral-8x7B, gemma4 31B, and gemma-4-26B-A4B-it.
Representative Data
- Llama-3.1-8B-Instruct: Increased from 472.28 t/s to 730.87 t/s at 256 prompt length, a 54.75% boost; the increase approaches 92% at 2048 prompt length.
- Mixtral-8x7B: Increased from 150.11 t/s to 470.41 t/s at 2048 prompt length, a 213.38% boost.
- gemma4 31B: Increased from 106.37 t/s to 222.32 t/s at 2048 prompt length, a 109.01% boost.
- gemma-4-26B-A4B-it: Relatively smaller improvement, around 14.20% at 2048 prompt length.
Conclusion
- ZenDNN significantly accelerates the preprocessing phase for long prompts.
- Such optimizations are biased towards inference stacks and local deployment performance, rather than changing the model's inherent capabilities.
More from Infra
- Anthropic S-1: training infra edge cut per-task cost 40% and doubled revenue growth — BenBajarin · 2026-10-26
- Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute — heghbalz · 2026-10-26
- US states halt new data centers as 72% of Americans oppose local builds — YvesMulkers · 2026-10-26
- $5M Seed Round Can't Afford One DeepSeek Training Run: GPU-less AI Startup's Three Paths — FinanceYF5 · 2026-10-26
- HBM demand shifts: low-stack inference plans shelved, Rubin to go 12-high only — BenBajarin · 2026-10-26
- CircuitHub unveils The Grid, a factory-scale robot making PCBs in the US twice as fast as Shenzhen — garrytan · 2026-10-26