Acceptance Is Not Speed: Length Inflation in Training-Free Lossy Speculative Decoding
Chiatzen Wang, Bill Xu, Haochen Gu, Hongyang Zhang
NAACL 2027
cs.CL
2026-10-17
Across 360 cells, five training-free lossy rules accept 1.34× the tokens per round but write 1.54× longer text, lifting mean verifier rounds to 1.10×.
Speculative decoding splits proposing from checking. A drafter writes a few tokens ahead, the target scores that block in one forward pass, and the longest accepted prefix is committed. Lossless verification keeps the target distribution, so completion length stays put and tokens accepted per round are a real speed metric.
Lossy verification keeps tokens that lossless decoding would reject. Training-free rules, CACTUS, mentored decoding, fuzzy speculative decoding, and speculative cascades, do this with a relaxation parameter α, and the speed they report is almost always accepted tokens per round or tokens per second. Once the distribution changes, length is no longer guaranteed. A faster cashier does not help if the cart itself gets longer. That lengthening is length inflation.
The round itself is unchanged. The drafter samples a block of 6 tokens; the target scores the block and one extra position in a single forward. A lossy rule swaps up to three distributions: πrej in the accept test, πres for the residual draw after a rejection, and πbonus for the extra token when the whole block is kept. Depending on the rule, α is a divergence budget, a threshold, a deferral cost, or a mixture weight, so α cannot be compared across rules. The comparison matches mean accepted length ℓ̄.
The five rules split into two families. CACTUS and mentored decoding boost the drafted token while limiting how far the distribution moves from the target p. The fuzzy rule and the two cascades, speccascopt and speccasctok, are switches: pass a per-position test and the draft is trusted. speccasctok is not strict even at α = 0. It can reject tokens that lossless verification would have kept.
λ is relaxed completion length divided by lossless length. The gain in tokens committed per round is the ratio of (ℓ̄+1) on the two sides, and the ratio of verifier rounds is about λ divided by that gain. If acceptance grows slower than λ, the target runs more forwards. The gain tops out at 7 divided by the lossless tokens per round, because a block holds only 6 draft tokens.
The targets are GPT-OSS-20B, Qwen3-8B, Llama-3.1-8B-Instruct, and DeepSeek-R1-Distill-Llama-8B. Eight dedicated heads, EAGLE-1, EAGLE-3, P-EAGLE, and DSpark, draft from the target hidden states. Three standalone models, Qwen3-0.6B, Qwen3-1.7B, and Llama-3.2-1B-Instruct, draft from tokens only. Twelve pairs, six benchmarks, and five rules make 360 cells: GSM8K, AIME24, HumanEval, LiveCodeBench, MT-Bench, and LongBench-v2. Everything runs on vLLM 0.26.0 on one H100, temperature 1.0, top-p 1.0, one sample per case. The main table uses each rule's loosest tested α.
At that setting the mean acceptance gain, the ratio of tokens committed per round (ℓ̄+1), is 1.34×, and mean completion length is 1.54× lossless. Median λ is only 1.08. Completions lengthen in 248 of 360 cells. The median moves more with the rule and the drafter than with the task: 0.97 for speccasctok versus 1.39 for cactus, 0.94 for standalone drafters versus 1.23 for EAGLE heads, and only 1.04 on GSM8K versus 1.14 on HumanEval.
| Rule | Mean λ | Mean round ratio |
| mentored decoding | 1.21 | 0.98 |
| cactus | 2.51 | 1.31 |
| speccascopt | 1.55 | 1.01 |
| rfuzzy | 1.46 | 1.31 |
| speccasctok | 0.96 | 0.86 |
| All five | 1.54 | 1.10 |
The round ratio has median 0.93 and mean 1.10. Spearman correlation with λ is 0.78, and with acceptance gain it is -0.12. Feeding the measured length into the correction predicts rounds within about 2% at the median. On the primary pairs the median wall-clock ratio is 1.01, and 11 cells save rounds without saving time, because a round costs more when it commits more tokens.
Accuracy drops by at least 2 points in 198 of 300 graded cells. Mean drops are 0.19, 0.14, and 0.21 for cactus, speccascopt, and rfuzzy, 0.03 for mentored decoding, and 0.005 for speccasctok, whose acceptance gain is only 1.11×. The 144 graded cells whose λ interval sits above 1 lose 0.21 on average; the other 156 lose 0.03. Code is worse: median losses are 0.09 on HumanEval and 0.11 on LiveCodeBench.
With an EAGLE head, the three aggressive rules use fewer rounds than lossless in 40 of 105 graded cells, and 35 of those 40 lose accuracy. cactus cuts rounds to 0.44× on R1-Distill-Llama-8B / LiveCodeBench, and accuracy falls from 0.58 to 0.02. On Qwen3-8B / AIME24, rounds fall to 0.55× and accuracy from 0.70 to 0.23. On the primary pairs, settings that stay within two points of lossless have a median round ratio of 0.96.
The drafter sets the inflation. Llama-3.1-8B-Instruct grows 3.30× with its EAGLE-3 head and 1.10× with Llama-3.2-1B. Qwen3-8B grows 1.18× with EAGLE-3 and 0.90× with Qwen3-1.7B. For EAGLE heads the λ interval is above 1 in 132 of 210 cells. On Llama with EAGLE, lossless answers already run 280 to 4,840 tokens, and cactus multiplies them by 4.0 to 13.2. Standalone Qwen3 drafters, which sample close to the target, do not lengthen significantly in any of their 60 cells and cut rounds by 10% to 46%. DSpark and P-EAGLE lose accuracy in 18 of 25 and 24 of 25 graded cells.
The extra characters go into the reasoning trace. On GPT-OSS-20B the analysis channel takes at least 92% of them in 25 of 30 cells. On Qwen3-8B more than 90% land inside think blocks at the median. Llama-3.1-8B-Instruct has no thinking trace, so the largest inflation lengthens the answer itself. Draft tokens that only the relaxed rule accepts are 7% to 24% of committed drafts. On the four rules other than speccasctok, their median target probability is 0.01 to 0.09, against 0.78 to 0.93 for tokens both rules accept. Raising the lossless sampling temperature to 1.5 inflates GPT-OSS-20B on GSM8K by 3.9× and drops accuracy to 0.31. Unlikely tokens lengthen generation on their own.
Gentler α does not remove it. On the seven pairs matched on ℓ̄, median λ is 1.07 at the gentlest setting and 1.22 at the loosest, and the gentlest setting needs more rounds than lossless in 97 cells against 94. At Qwen3-8B's recommended temperature of 0.6, speccascopt still inflates GSM8K and LiveCodeBench by 1.5×, while mentored decoding, cactus, and speccasctok stop inflating on those two sets.
Forcing the reasoning trace to close near the budget recovers answers. Over 1,132 such runs, correct answers rise from 126 to 439: 335 flip from wrong to right, 22 from right to wrong. On GSM8K, 50% (GPT-OSS-20B) and 82% (Qwen3-8B) of the extra tokens come after the reference answer is already written. On AIME24, GPT-OSS-20B reaches the answer at median token 3,865 instead of 1,721, and at least 59% of the extra tokens come first.
EAGLE heads already ship in vLLM, SGLang, and TensorRT-LLM. A report that only prints tokens accepted per round will call these lossy rules faster. Mean length here is 1.54× and mean acceptance gain is 1.34×, so mean verifier rounds rise to 1.10×. Acceptance is not speed once verification is relaxed. Length and round count have to be reported beside it.
speccasctok relaxes the least, stays closest to lossless accuracy, and uses 0.86× the rounds. Standalone Qwen3 drafters that stay near the target cut rounds with a smaller accuracy bill. The setting that works is choosing which draft tokens to keep.
No shared ℓ̄ is reachable by every rule, so the main table compares each method at its own loosest α, four published values apiece. Relaxed runs hit the token budget more often, measured length is right-censored, and the inflation is understated. The GSM8K cap of 2,048 tokens is tight for Qwen3-8B: 25% of its lossless runs already hit it, and some of the accuracy loss is an unfinished answer. Each case is sampled once. AIME24 has 30 problems. Wall-clock time is client-side at batch size 1 on the primary pairs only. MT-Bench is one judge's first-turn score.
The rules are reimplementations in vLLM's rejection sampler, checked when they fall back to strict verification, not against the authors' code. On some pairs only the acceptance test changes. Relaxed draft length is fixed at 6. Why a token the target itself would rarely sample makes a reasoning model think longer is left open.
The worst inflation is on Llama-3.1-8B-Instruct, which has no thinking trace, so the extra text is the answer. Answers recovered by closing the trace early concentrate on Qwen3-8B, with little recovery on AIME24.