Speculative Decoding Explained: How Draft Models Yield k+1 Tokens Per Pass Without Changing the Distribution

cneuralnetwork · x · 2026-10-22

A well-illustrated thread walking through speculative decoding, one of the neatest tricks in LLM inference.

Why decoding is slow: autoregressive generation needs n sequential forward passes; at small batch sizes it's memory-bandwidth bound.
Key insight: generation and verification are different computational problems — given k tokens, causal attention lets the transformer verify them all in one parallel forward pass.
Flow: a cheap draft model q proposes k candidate tokens; the target model p verifies them in parallel.
Rejection sampling preserves the distribution: accept x with probability min(1, p(x)/q(x)); on rejection, sample from the residual r(x) ∝ max(0, p(x)−q(x)). Since min(p,q)+max(0,p−q)=p, the final token distribution is exactly the target's — no retraining, no approximation.
Details: only the consecutively accepted prefix is kept; if all k tokens pass, one extra token can be sampled, so a single target pass commits up to k+1 tokens.
Speedup bound: expected acceptance α = Σₓ min(p,q) = 1 − tv(p,q); closer draft-target alignment helps, but a larger draft slows drafting — the core tradeoff.

More from Infra