2026-10-24
SpecTr-GBV checks K independent drafts as blocks. On DeepSeek-33B, mean speedup is 2.12× versus 1.64× for standard speculative decoding, and verification drops by over half.
Autoregressive decoding emits one token per forward pass of the target model. Speculative decoding lets a small draft model propose tokens and the large model accept a prefix of them, so accepted tokens are not recomputed. A single draft often dies at the first mismatch, and everything after it is thrown away. Few tokens survive the round.
Two fixes have stayed separate. SpecTr draws several drafts that do not depend on each other and, at each position, uses optimal transport to accept one token. More candidates help, but verification is still position by position. If every candidate at one column is rejected, the rest of the drafts are discarded. Sun and colleagues showed that this does not maximize the expected number of accepted tokens in one round, and they gave greedy block verification for a single draft: score prefixes as blocks and keep the longest one that passes. The block gets longer. There is still only one candidate stream.
SpecTr-GBV puts multiple drafts and block acceptance into one optimal-transport problem. Given a prefix, the draft model samples K sequences of length L, independent and identically distributed. Acceptance length is the largest i such that some draft's first i tokens match the target prefix exactly. The objective is to maximize the expected length over couplings that stay conditionally independent given the target sequence. That constraint matches how the drafts are actually produced: each sequence is sampled on its own and is not rewritten to agree with the others.
The transport problem is not solved directly. Joint distributions over sub-blocks are unavailable, there is no closed form once K is greater than 1, and a linear program over the vocabulary is too expensive. The algorithm walks the drafts one by one and does greedy block verification on each. From the length already accepted, it computes an acceptance probability for each longer sub-block and draws a uniform random number. A longer accepted prefix updates the record. Rejected sub-blocks go into a set and are skipped if they show up again, so the same block is not resampled. The longest accepted sub-block is kept. If the full length L passes, one extra token is drawn from the target. If it does not, one token is drawn from a residual distribution that removes probability mass the drafts already covered.
Before the next round, the target distribution at positions that were not accepted is rewritten. That step is what keeps the output distribution equal to ordinary sampling from the target model across rounds. Speed is not bought by changing the distribution. With K fixed, the procedure meets the upper bound on expected acceptance length inside i.i.d. draft generation. The bound rises strictly as K grows, approaches L as K goes to infinity, and collapses to ordinary greedy block verification at K = 1. Each selection costs on the order of the vocabulary size. SpecTr's K-SEQ approximation also binary-searches a scale factor, which adds a log K term.
The main setup uses DeepSeek-33B as the target and DeepSeek-1.3B as the draft, the DeepSeek-Coder family in the citation. Draft length is 12, temperature 0.4, K = 3, and each of five datasets uses 1,000 prompts. Block efficiency is the average number of tokens settled per serial call to the target. Speedup is autoregressive wall time divided by the method's wall time.
| Method | Mean block efficiency | Mean speedup |
| Autoregressive | 1 | 1 |
| Standard speculative decoding | 7.64 | 1.64 |
| SpecTr | 8.39 | 1.96 |
| Greedy block verification | 7.83 | 1.67 |
| SpecTr-GBV | 8.59 | 2.12 |
Against standard speculative decoding, mean block efficiency is 12.4% higher and speedup is 29.3% higher. Against SpecTr, block efficiency is 2.3% higher and speedup is 8.2% higher. Against single-draft block verification, block efficiency is 9.7% higher and speedup is 27.0% higher. On HumanEval, speedup moves from SpecTr's 2.21 to 2.50. On GSM8K it moves from 2.26 to 2.46. Standard deviations on those cells run from 0.30 to 0.58, the same scale as the gaps. The paper does not report a significance test. The five-dataset mean is the more stable summary.
The wall-clock split shows that the gain is not mostly extra accepted tokens. On GSM8K with the 33B target, total time versus SpecTr falls from 19.24 s to 18.03 s, verification from 1.01 s to 0.47 s (−53.5%), acceptance rate from 0.521 to 0.555, and throughput from 27.48 to 29.64 tokens/s. On HumanEval, verification time drops 74.8%.
Longer drafts keep raising block efficiency, then speedup turns down. On HumanEval with the 33B model, K = 3 and temperature 0.4, length 20 reaches speedup 2.87 and length 24 falls to 2.68, while block efficiency rises from 14.58 to 16.52. Draft compute eats the calls saved on the target. Acceptance rate rises from K = 1 to K = 7, in line with a bound that increases in K. Temperatures 0.1, 0.4, and 0.7 barely move either metric.
With a 6.7B target, the same 1.3B draft, and length 8, mean speedup across the five datasets is only 1.20, and Alpaca is 1.03. CodeLlama-13B with a 7B draft averages 1.42. Vicuna-13B with a 7B draft averages 1.30. On the one-billion-word benchmark, standard speculative decoding with Vicuna has speedup 0.91, slower than plain autoregression, and SpecTr-GBV only reaches 1.01.
Multi-draft search and block verification used to be separate budgets. When drafts must be sampled independently, the two can hit the optimal expected acceptance length of that setup, and verification is cheaper than SpecTr. What matters in deployment is the gap between target and draft. A 33B target with a 1.3B draft lands near 2.1× wall time. A 6.7B target with the same draft often keeps barely more than 10%, and on some datasets almost nothing. Longer is not automatically better. On this HumanEval curve, speedup peaks near draft length 20.
The optimality claim is inside i.i.d. drafts. Tree drafts, retrieval drafts, and drafts rewritten from earlier context sit outside the bound, and the paper does not compare against systems such as SpecInfer or Sequoia. The block-efficiency gain over SpecTr is a few percent, and on several individual datasets it sits inside the standard deviation. A good share of the speedup comes from dropping the log-factor search and from a slightly higher acceptance rate, not from a much longer accepted block. Output quality is guaranteed by the distribution-preservation theorem. The tables do not report HumanEval pass rate or GSM8K accuracy. At nonzero temperature, single samples from unbiased methods still differ. The models are DeepSeek-Coder, CodeLlama, and Vicuna, not the draft heads, such as EAGLE, that show up in current serving stacks.