Speculative decoding math explained: acceptance rate equals 1 minus TV distance
techNmak · x · 2026-11-04
A systematic walkthrough of the math behind speculative decoding: a small draft model q proposes K tokens sequentially and the large target model p verifies them in one forward pass. Each token is accepted with probability min(1, p/q); on rejection, a correction is sampled from the normalized positive part of p−q. Accepted proposals contribute min(p,q) and corrections contribute max(p−q,0), which sum exactly to p — so the output distribution is preserved exactly. The expected acceptance rate is α = 1 − TV(p,q), meaning a draft's confidence alone doesn't determine acceptance. With K=3 and α=0.7, expected tokens per round is 2.53. Engineering-wise, K selection matters: more speculative tokens can increase per-round output while making the whole system slower due to drafting cost, verification overhead, KV-cache and hardware factors.
More from Research
- DeepMind's Yasaman Bahri to lecture at Harvard on how language statistics shape LLM representational geometry — yasamanbb · 2026-11-04
- Allen AI benchmark clarified: long-context eval for state-tracking tasks — lambdaviking · 2026-11-04
- Probe Robustness Paper Wins Spotlight at EMNLP's UncertaiNLP Workshop — PMinervini · 2026-11-04
- Meta and Duke paper: reward hacking spreads between agents, hitting Claude and Gemini 73% of the time — rohanpaul_ai · 2026-11-04
- UT Austin professor adds hands-on Transformers chapter to free ML e-book — GeostatsGuy · 2026-11-04
- Intology's Locus autonomously adapts frontier science AI models to proprietary data — anshulkundaje · 2026-11-04