Speculative Decoding Explained: a 30% Wrong Guesser Never Changes the Output
anbarth · x · 2026-10-26
Antje Barth walks through the core guarantee of speculative decoding in a code-backed GitHub post: even a deliberately bad guesser (30% of guesses thrown out) cannot change the target model's output — only speed differs. The key is Leviathan, Kalman & Matias' 2022 rule: with guesser distribution q and target distribution p, keep a sampled token with probability min(1, p/q), otherwise resample from p − q; that one subtraction makes the output exactly the target's for any guesser. Context: vLLM's docs (updated Oct 23) now list the lossy FLy Verification beside the exact checker, and SGLang exposes thresholds that accept more aggressively. Includes working code.
Related event: AWS Explains Why Speculative decoding Never Changes LLM Output(2 posts)→
More from Infra
- Anthropic built Scion to replace git as AI-driven merges jumped from 2,500 to 30,000/day — bibryam · 2026-10-26
- NVIDIA shows RTX Spark powering sim-to-real robotics: Microduck robot learns somersault in simulation — Aiden_Tech_Ai · 2026-10-26
- MiniMax H3 community adds 8-step reference control, 4-step distill, Apple Silicon support — Hailuo_AI · 2026-10-26
- SPCX acquires Modal, per unconfirmed one-line report — arthurcolle · 2026-10-26
- Guess the era: multimode graded-index glass waveguide optical backplane — jwt0625 · 2026-10-26
- Stanford and UC Berkeley's TokenGate speeds long-context prefill by up to 11.3x — MundanePercentage674 · 2026-10-26