WingLLM: Go daemon doubles 7B LLM speed to 2.54 tok/s on a 2GB GPU via speculative decoding
rbm3267 · reddit · 2026-11-17
A developer stuck at 1.29 tok/s running 7B models on a 2GB Quadro P620 diagnosed the bottleneck as memory bandwidth, not CPU compute — streaming 4.7GB of weights per token caps DDR4 systems at 1.5-2 tok/s. His fix: heterogeneous speculative decoding, where a 397MB 0.5B draft model on GPU proposes 3-4 tokens and the CPU-resident 7B verifies them in one pass, with 75-82% acceptance on code. Real result: 2.54 tok/s, a 1.97x speedup with identical output.
He open-sourced WingLLM, an 11MB Go binary (MIT) that auto-pairs compatible tokenizers, scans local Ollama manifests, auto-pulls draft models, one-click connects to VSCode Continue/Cline, embeds a web chat UI, and ships an MCP server for Claude Desktop/Cursor. Install scripts for Windows/macOS/Linux are on Hugging Face.
More from coding & agent
- Qwen4-Max goes live on Lebrel, tops DeepSWE coding benchmark at 7% below list price — Alibaba_Qwen · 2026-11-17
- Pageloop Launches a Docs Agent That Auto-Captures Screenshots and Updates Stale Docs — prasanna_says · 2026-11-17
- Developer tip: use MCP to pass UI templates alongside fetched data, not just raw results — js_craft_hq · 2026-11-17
- OutfitterBench tests LLM agents on 200-day rental business, exposing commitment failures — Jie Huang · 2026-11-17
- Craftax framework evolves code policies via feedback cascade, reaching final stage 50% of time — Víctor Gallego · 2026-11-17
- Entity disambiguation: sentence context hits 97.9% vs 63.5% for name matching — YvesMulkers · 2026-11-17