llama.cpp WebGPU adds Q2_0 support, Ternary Bonsai prefill jumps ~50x to 2000 t/s
pmttyji · reddit · 2026-10-22
PR #30914 by teddygood adds Q20 quantization support to llama.cpp's WebGPU backend, benchmarked on an RTX 3060 with native Dawn WebGPU:
- Ternary Bonsai 1.7B Q20: pp512 up from 35 t/s to 1915.91 t/s; tg128 from 14 t/s to 94.75 t/s
- Bonsai 2 2B Q20: pp512 up from 36.4 t/s to 2039.89 t/s; tg128 from 12.57 t/s to 71.72 t/s
The change is small (+81/-1 lines, 5 files) but turns a nearly unusable quant format on WebGPU into 50-55x faster prefill throughput — a concrete win for running small quantized models in browser/WebGPU environments.
More from Infra
- Dev shows off self-built infra stack — HTTP clients, attention kernels, model routers — to quit dependencies — wavefnx · 2026-10-22
- Can a $100 MacBook Pro M2 24GB serve local LLMs for Home Assistant? — Ejo2001 · 2026-10-22
- Tune your threshold before adding an LLM judge: optimal escalation rate was 0% — Reasonable_Royal_621 · 2026-10-22
- Where your $20 Claude subscription goes: an interactive AI value-chain explainer — aishashok14 · 2026-10-22
- CVE-2026-111943: Compromised intermediate certificate authority discovered — keks24 · 2026-10-22
- AI's hidden storage bill: an 8-second video leaves 500MB behind — MIT Tech Review AI · 2026-10-22