2-bit Drafting with In-Memory Residuals Enables Speculative Decoding on Small GPUs

An open-source approach, twobit-residual-spec, keeps a 2-bit copy of the model on GPU as the drafter with residual weights in system memory, enabling speculative decoding for models that don't fit in VRAM.

2026-10-22 ~ 2026-10-22 · 2 related posts

• 2-bit draft in VRAM + exact residual in RAM: speculative decoding for models that don't fit your GPU — Happy_Guitar3521 · 2026-10-22
• 2-bit draft in VRAM + residual in RAM: speculative decoding when the model doesn't fit — Happy_Guitar3521 · 2026-10-22