2-bit draft in VRAM + exact residual in RAM: speculative decoding for models that don't fit your GPU

Happy_Guitar3521 · reddit · 2026-10-22

The author shares an unfinished speculative decoding idea for cases where a model's Q8 doesn't fit in VRAM (e.g., 27B+ models on 16-32GB cards):

Design: keep a 2-bit copy on GPU as the drafter and the residual R=W−Ŵ (6-8 bit) in system RAM; verification splits Ŵx on GPU and Rx on CPU with only activations crossing PCIe, re-scoring the whole candidate tree in one batched pass.
Measured results (Qwen3.5-4B, Gemma 4 12B/31B, greedy): the verifier is more precise than Q80 (about half the KL vs bf16); the 2-bit draft reading the verifier's exact KV cache beats Google's official Gemma 4 drafter by 25-30% tokens accepted per check; a CPU kernel makes verifying 16 tokens cost about the same as 1.
Key tricks: calibrating on chat data instead of C4 boosted draft acceptance ×1.37 (1.22→3.03 tokens on Polish prompts); a uniform grid beat Lloyd-Max for the residual.
No engine was built; a complete POC recipe is included. If Q8 fits in VRAM, just use llama.cpp.

More from Infra