Residual sampling: why speculative decoding preserves the exact target distribution
cneuralnetwork · x · 2026-10-22
Part six covers what the author calls the cleverest step — sampling from a residual distribution upon rejection:
- Define r(x) ∝ max(0, p(x) − q(x));
- Accepting from the draft contributes min(p, q); residual sampling contributes max(0, p − q);
- Summing: min(p,q) + max(0, p−q) = p, so the final token distribution is exactly the target distribution — the mathematical guarantee that speculative decoding never changes model output.
Related event: Rejection and Residual Sampling Preserve Output Distribution(2 posts)→
More from Research
- Steering a single neuron in Olmo 3 bypasses post-trained assistant behavior — Sauers_ · 2026-10-22
- NVIDIA's Nemotron-Personas-Germany dataset trends on Hugging Face — nvidia · 2026-10-22
- CMU paper defines solver-sufficient symbols to cut hybrid robot planning cost — Yiyuan Pan · 2026-10-22
- NeurIPS Creative Track 2026 seeks sponsors to fill venue with AI art installations — dribnet · 2026-10-22
- JHU researchers preview their NLP papers coming to EMNLP 2026 in Budapest — mdredze · 2026-10-22
- Google DeepMind's Discovery team makes new progress on the elusive 6th Busy Beaver number — robertarail · 2026-10-22