Speculative Decoding in One Subtraction: Why Even Backward Guessers Preserve Output
anbarth · x · 2026-10-26
Antje Barth breaks down the math of speculative decoding: a small model guesses tokens, a big model verifies them. The share of guesses kept equals the overlap between the two probability distributions — a perfect guesser keeps everything, while a uniform guess (equal chance across Qwen3-8B's 151,936-token vocabulary) keeps almost nothing.
The striking takeaway: even a guesser with every chance backwards still gets 30% of guesses thrown out, yet the output is exactly identical to the big model's — provable with a single subtraction. The original post includes a step-by-step walkthrough with code.
Related event: AWS Explains Why Speculative decoding Never Changes LLM Output(2 posts)→
More from Infra
- Reddit user builds $3360 Threadripper box with 256GB DDR4 to run large LLMs locally — sitytitan · 2026-10-26
- Top 1% of AI spenders direct over a third of AI budgets to raw GPU compute, Ramp data shows — rohanpaul_ai · 2026-10-26
- Running Qwen3.8B across 3 GPUs and 130GB RAM only hits 18 T/s — MageLD · 2026-10-26
- Used RTX 3090s Now Top $1500 on eBay as Owner Hunts a Replacement — like_a_tensor · 2026-10-26
- Ternary 1.58-bit Nemotron 30B-A3B GGUF Runs 368K Context on 12GB GPU — pmttyji · 2026-10-26
- DeepSeek V4.1-Flash hits 49 tok/s decode on dual Radeon R9700 in local test — Longjumping-Elk-7756 · 2026-10-26