2-bit draft in VRAM + residual in RAM: speculative decoding when the model doesn't fit
Happy_Guitar3521 · reddit · 2026-10-22
A developer shares an open-source inference scheme (twobit-residual-spec) he already uses in production: keep a 2-bit copy of the model on GPU as the drafter and the 6-8 bit residual weights in system RAM as the verifier; the GPU handles the 2-bit part, the CPU the residual, and only activations cross PCIe, so the draft costs no extra VRAM.
Measured (dequantized weights in PyTorch, no tok/s yet): 2-bit + 8-bit residual is closer to bf16 than Q80 at about half the KL; a 6-bit residual reaches Q80-level quality while reading 28% fewer bytes from RAM; the 2-bit drafter reusing the verifier's KV cache accepts 25-30% more tokens per check than Google's official Gemma 4 drafter (31B, 16-node tree: 5.23 vs 4.20); the CPU kernel checks 16 tokens in about the time of one since it's memory-bound. Closest work is SubSpec, which uses a separate 4-bit copy streamed over PCIe. Only makes sense when Q8 doesn't fit in VRAM.
More from Infra
- MiniMax M3.1 Flash serving picks vLLM, with Inferact as API launch partner — woosuk_k · 2026-10-22
- Why Nvidia keeps its market share: deep systems integration and Dell partnerships ASIC rivals lack — PTrubey · 2026-10-22
- One Cloudflare Durable Object at $9/month replaces Supabase's ~$33K real-time sync plan — threepointone · 2026-10-22
- SF Systems Meetup: Agents-Only Version Control and Postgres Rewritten in Rust — ShadajL · 2026-10-22
- Cloudflare saw thousands of php2shell exploit attempts before disclosure, millions now — ivan_bezdomny · 2026-10-22
- Jensen Huang: NVIDIA Is "Literally the Only One" You Can Trust With a $100B Blind Chip Order — jfischoff · 2026-10-22