Qwen3.5-2B at 620 tok/s in Chrome: model-specific WebGPU kernels, 6.6× WebLLM
tixiren · reddit · 2026-11-03
Developer tixiren released webyIQ, a per-model-specialized browser LLM engine: Rust compiled to wasm plus hand-written WGSL kernels for Qwen3.5-2B (Gated DeltaNet + gated attention) and MiniCPM5-2B. Benchmarks (same model, same prompts, greedy, Chrome 156): 620 tok/s generation on RTX 5080 (417 without speculation) vs WebLLM 0.2.87's 94 and transformers.js 4.3's 31; 0.20 s first response on 1k-token prompts (WebLLM 0.81 s, transformers.js 129 s); 37 tok/s on an AMD 2-CU iGPU. Key techniques: fused int4 (GPTQ, group 32) × int8 GEMV/GEMM with dot4I8Packed and subgroups; a decode loop that stays on GPU (argmax on GPU, pipelined readback); speculative decoding via Qwen3.5's built-in MTP head with GPU-verified drafts (bit-identical to greedy, auto-disabled where it doesn't pay); weights at only +2.4% WikiText perplexity vs bf16, 1.1 GB. Side effect: a 2B model can drive web pages through the DOM at 0.2–0.4 s per step, with a browser-agent demo and a Chrome/Edge side-panel extension. Apache-2.0 prototype, tested on Windows only.
More from coding & agent
- Dev proposes LLM-generated ultra-rare content to build anime-style MMORPGs — rickasaurus · 2026-11-04
- Dev builds Chrome extension to bring back the classic Twitter bird on X — joonasvirtanen · 2026-11-04
- Developer lets his app record 'wishes' and auto-spawn OpenCode background agents to fulfill them — SubtleGradient · 2026-11-04
- Post-train your model on your own agent harness with OpenHands, Harbor, SkyRL and Snowflake — StasBekman · 2026-11-04
- Google Antigravity redesign leaked with context usage and quota indicators — clmt · 2026-11-04
- Microsoft's 4B Mage-VL-Cowork Hits 66% on OSWorld-Verified, Near Human Level — microsoft · 2026-11-04