Int8 AVX512 VNNI quantization cuts CPU inference to 34ms vs 90ms fp32—2.6x faster
wavefnx · x · 2026-11-12
wavefnx reports on a custom ONNX Runtime pipeline: quantized Int8 with AVX512 VNNI runs in 34ms on CPU vs 90ms for default fp32—about 2.6x faster and 4x lighter, with only 1-2% semantic loss from quantization, which he calls "a clear win".
He also notes he has been building AMD-compatible runtimes, compat layers and kernels since the start of the year while everyone focused on Nvidia, and now waits to see if AMD can deliver under the spotlight.
More from Infra
- US Largest Public Pension Board Member Flags AI Investment Risk From Data Center Backlash — asusarla · 2026-11-12
- Agent harnesses are heading toward vertical integration, argues 0xblacklight — HankYeomans · 2026-11-12
- CoreWeave turns to customer prepayments and shorter contracts to fund AI growth — TechCrunch AI · 2026-11-12
- Foreign Investors Pull $50B From Asian Equities in October on AI Cash-Burn and Regulation Fears — nordicinst · 2026-11-12
- Bernstein Ranks Data Center Pipelines: META, MSFT, GOOGL Top the Credibility List — aronchick · 2026-11-12
- Photoroom reclaims ~1,000 idle GPU hours weekly using SkyPilot across Slurm and Kubernetes — skypilot_org · 2026-11-12