D3D overlaps diffusion drafts with verification for 5.71× on Qwen3-32B

D3D: Decoupled Diffusion Drafting for Asynchronous Speculative Decoding

Baoyou Chen, Ruiqiao Mei, Guancong Lin, Tao Song, Weihao Yuan, Qun Liu, Siyu Zhu

cs.AI, cs.LG

2026-11-04

D3D trains a standalone diffusion drafter that drafts the next block during verification, averaging 5.00× on Qwen3-8B and 5.71× on Qwen3-32B versus 4.03× and 3.67× for DFlash.

What problem this solves

Speculative decoding asks a smaller model to propose several tokens and the target model to check them in parallel. Rejection sampling keeps the output distribution identical to decoding with the target alone. The speedup depends on how many tokens are accepted, how long drafting takes, and how long verification takes.

DFlash writes a whole candidate block in one diffusion pass, but the next block needs fresh hidden states from the target. Drafting cannot start until verification has produced those states. Positions inside a block run in parallel. Rounds do not. A deeper drafter usually accepts more tokens and also spends longer, so the extra acceptances get spent on draft latency.

Method

D3D is a standalone diffusion drafter. At inference it never reads target hidden states. It encodes the committed token prefix itself and predicts one block in a single forward pass. While the target is still verifying, one packed pass prepares successor blocks for mid-block rejections and for full acceptance.

Training starts from a pretrained autoregressive model. Prompts come from Open-PerfectBlend, 1.42 million instructions, and responses are regenerated by the matching target in non-thinking mode. Target-aligned training runs for 3 epochs, about 360 GPU-hours. Anchors are sampled along each response. The anchor token stays visible, the next B-1 positions are masked, and bidirectional attention inside the block predicts them together. The prefix is encoded once with causal attention and shared across packed blocks. Verification stops at the first rejection, so position weights follow DFlash's exponential decay and put more loss on earlier tokens.

An auxiliary autoregressive loss keeps next-token prediction on the unmasked response. It is added to the denoising loss with λ = 1. Denoising alone lets the pretrained next-token behavior fade. With λ = 1, acceptance length rises by 0.24 to 0.40 tokens over λ = 0 on four benchmarks. λ = 1.5 adds nothing further.

Layer pruning ranks layers by acceptance length, the number of tokens a round actually keeps, not by language-model loss. Each round drops the layer whose removal hurts validation acceptance the least, distills from the frozen unpruned drafter, then scores the layers again. The teacher distribution is mixed slightly toward the supervised token, so a confident wrong mode in the teacher does not pull the student off the label. With Qwen3-8B as the target, dropping 5 of 36 layers keeps more than 99% of mean acceptance length on MATH-500, HumanEval, and MT-Bench, and lifts mean speedup on those three from 4.44× to 4.69×. The 31-layer drafter is the fastest depth tested. Cutting further lowers both acceptance and speedup.

The main setup uses block size 16, position coverage 12, and 2 alternate anchors at each covered rejection point, packing 384 token positions. Alternate anchors are the drafter's next two most likely tokens after the proposed one. A cache hit reuses the rest of that branch. A miss forces a synchronous redraft before the next verification round. When packed drafting is no slower than target verification, that round's draft latency is hidden.

Results

The drafter is D3D-8B, initialized from Qwen3-8B and pruned to 31 layers, trained separately on responses from each target. Targets are Qwen3-8B and Qwen3-32B. Seven benchmarks cover math, code, and chat, all with greedy decoding. Acceptance length τ includes the bonus token the target appends each round. DFlash, PARD-2, and D3D use block size 16. EAGLE-3 and DSpark use 8.

TargetMethodMean speedupMean τ
Qwen3-8BD3D5.00×7.88
Qwen3-8BDSpark4.44×5.59
Qwen3-8BPARD-24.08×5.62
Qwen3-8BDFlash4.03×5.96
Qwen3-8BEAGLE-32.88×4.56
Qwen3-32BD3D5.71×7.73
Qwen3-32BDSpark3.92×5.48
Qwen3-32BPARD-23.88×5.38
Qwen3-32BDFlash3.67×4.82
Qwen3-32BEAGLE-33.46×5.08

On both targets, D3D's τ is longer than these methods and SSD on every benchmark. The peak is GSM8K with Qwen3-32B, 7.48× at τ = 9.58, against DFlash at 4.46× and τ = 5.72. On Qwen3-8B, D3D has the highest speedup on five benchmarks. LiveCodeBench ties DSpark at 4.44×. On MT-Bench, DSpark reaches 3.17× and D3D 2.68×, even though τ is 5.71 versus 3.82. On Qwen3-32B MT-Bench the speedup edge over DFlash is 3.01× versus 3.00×, while τ is 5.65 versus 3.89. Chat has the shortest τ and the weakest speedup, so the extra accepted tokens do not show up as the same gain in wall-clock time. SSD, also asynchronous, averages only 1.84× on Qwen3-8B.

Concurrency is measured with the smaller D3D-1.7B drafter and Qwen3-32B. MATH-500 moves from 154 tokens/s at one request (6.1×) to 864 tokens/s at 16 requests (2.7×). HumanEval and MT-Bench show the same pattern: throughput up, relative speedup down.

Pretrained initialization finishes near τ ≈ 9.8 on GSM8K. Random initialization finishes near 4.5. On AIME25, coverage 12 and fan-out 2 give about 5.0× at a 71.3% cache hit rate. Raising coverage to 16 lifts the hit rate to 78.3% and lowers speedup, because every packed pass pays for branches verification never takes.

Why it matters

Rejection sampling stays in place, so the output distribution is still the target's. The schedule is different: the drafter no longer waits on target hidden states, and the next block can be prepared while the current one is verified.

A larger target makes verification more expensive, so the same draft is easier to hide. Against the strongest mean speedup in the table, the gain is 12.6% on Qwen3-8B (5.00× versus DSpark at 4.44×) and 45.7% on Qwen3-32B (5.71× versus 3.92×). A drafter cut from Qwen3-8B is a better match for a 32B target than for an 8B target.

Against DFlash this is an incremental systems result. Block drafting already existed. D3D removes the hidden-state dependency and prunes layers by acceptance length, which is what turns the longer τ into wall-clock speedup. PARD-2 can also run asynchronously without target features. On Qwen3-8B GSM8K, D3D reaches 6.71× against PARD-2 at 6.16×. On HumanEval the gap is 5.19× versus 3.95×.

Limitations

The hardware is not matched. D3D, SSD, and PARD-2 use two H800 GPUs, one for the target and one for the drafter. DFlash, DSpark, and EAGLE-3 run on a single GPU. There is no per-GPU throughput comparison, and no comparison against spending those two GPUs on tensor-parallel target inference with no drafter. Next to Qwen3-8B, a drafter that started as Qwen3-8B and lost only five layers is a real memory cost.

A cache miss falls back to synchronous drafting, and the overlap disappears for that round. The stack is Transformers, not SGLang or vLLM. Reported numbers use greedy decoding in non-thinking mode. Sampling and other model families are absent.

The claim of keeping over 99% of acceptance length covers only Qwen3-8B on three benchmarks. The headline 5.00× and 5.71× already use the 31-layer drafter. End-to-end speedup of the unpruned drafter on all seven benchmarks is not reported.

Terms

Source

What people are saying

Related papers

All paper explainers