ik_llama.cpp PR adds AVX2/VNNI kernels for Q2_0_R8, +45% prompt processing on CPU
Any_Bus9205 · reddit · 2026-11-01
PR #2785 in ikllama.cpp adds Q20R8, an 8-row repacked layout for Q20 tensors (2.25 bpw) with dedicated AVX2/AVX-VNNI kernels (dpbusd/maddubs), plus fixes for odd block counts in legacy Q20. It targets recent GSQ/RCO low-bit releases (e.g. ISTA-DASLab collections) and ternary-weight models.
Benchmarks (i5-13420H, Ternary-Bonsai-2-4B Q20):
- Native Q20: 34.46 t/s (pp512) | 9.56 t/s (tg128)
- Runtime repack (-rtr): 49.99 | 11.64 t/s
- Q20R8 offline repack: 49.08 | 11.55 t/s
- Net: +45% prompt processing, +22% generation, perplexity essentially unchanged (13.0721 → 13.0763)
Two workflows: run existing Q20 GGUFs with the -rtr flag for dynamic in-memory repacking at startup, or pre-convert with llama-quantize --repack into a Q20R8 file. The PR includes full build and test commands.
More from Infra
- Meta's $97.5B mega raise: $25B share sale 5x oversubscribed, Nvidia commits $3B, AMD $1B — Beth_Kindig · 2026-11-01
- Server CPU shipments to grow 31% in 2026, over 40% in 2027 on inference and agentic AI demand — Beth_Kindig · 2026-11-01
- "Family GPU" box shares compute over WiFi, runs 750B model at ~8 tok/s — takuonline · 2026-11-01
- Interactive visualizer lets you zoom into Esperanto's ET-SoC-1 AI chip, down to FMA instructions — yaroslavvb · 2026-11-01
- Musk: Colossus 2 to host ~1.2M Blackwell GPUs by end of year — scaling01 · 2026-11-01
- Local Maximus meme: six RTX 3090s, 256GB RAM, servant to no API — gklambauer · 2026-11-01