SpecPilot: Autonomous Algorithm Discovery and Systems Optimization for Speculative Decoding
Kunjun Li, Wenhao Chai, Yihao Liang, Yuezhou Hu, Shuyao Xu, Yi Pan, William X. Zheng, Kannan Ramchandran, Kurt Keutzer, Jiantao Jiao
cs.LG
2026-11-02
SpecPilot raises speculative decoding from 4.46× to 10.87× over autoregressive decoding at one request, and from 1.94× to 3.93× batched, with no training and lossless outputs.
Speculative decoding lets a small drafter propose tokens that the target checks in one forward pass. Parallel drafters such as DFlash2 and DSpark write 8 or 16 of those tokens in a single pass. With one request, loading the target weights dominates the step, so extra verify rows are almost free. On an H100, after the first optimization round, 80% of a 23.61 ms step on Qwen3.8-27B is weight traffic. At the chip's peak 3.35 TB/s, those reads are still 92% of the bytes moved in the step.
Released code, called stock here, verifies one greedy chain. In offline replay, DFlash2's 8-position chain accepts 4.25 tokens. An oracle that keeps accepting while the target token stays in the drafter's top-16 reaches 6.42 at 8 positions and 8.20 at 16. Batching moves the bottleneck. Each request brings its own attention, recurrent state, and prefill, and stock's speedup over autoregressive decoding roughly halves: FP8 Qwen3.8-27B falls from 3.40× at one request to 1.89× at 24. A low-precision pre-check that helps one request is 13% slower on BF16 batching.
SpecPilot separates the agents that edit the system from the controller that times it. Claude Opus 5.5 searches inside SGLang, once per deployment.
Speedup over autoregressive decoding is τ, tokens accepted per step, divided by the step-time ratio. The ceiling assumes the best tree, one exact verify pass, and every kernel class at its byte floor, the slower of 60% of peak dense compute and 90% of peak bandwidth. The largest of three factors sets the round: leftover tree acceptance, distance to the byte floor, and time outside the profile. One direction gets one GPU. Under 3% expected net gain, the agent does not write a new algorithm. Promotion requires the 95% lower bound on the gain to clear 1% plus twice the measurement noise, or about 3% if outputs diverge.
Losslessness is structural. Every accepted token has to sit on the argmax path of a forward pass over the exact weights. Token forcing scores outputs token by token on stock with an FP32 LM head. Stock already mismatches 0.72% of tokens from rounding. A mismatch rate more than 0.5 points above stock, or a rise in large-margin events, rejects the patch. Task accuracy misses the same fault: after a logit bias of 2 is added to every 7th vocabulary id, the accuracy interval still covers zero. Promoted mechanisms go into a memory of 140 entries. A reused entry is retuned and retimed, and is not counted as a new find.
W12 packs each BF16 weight into 12 bits and checks the decoded tensor against the checkpoint at load. Weights in one output channel usually occupy a few powers-of-two bins, so each channel keeps a 16-bin window and stores a sign, a 4-bit exponent offset, and all 7 mantissa bits. Values outside the window go to a sparse escape list. Converted layers of Qwen3.8-27B shrink from 44.67 GB to 33.58 GB. Decoding happens in registers. A denser 11-bit variant took about 3× as long to decode and was dropped. On H100 the LM head stays BF16. An INT8 copy with an error bound throws away tokens that cannot win, cutting head latency from 878 μs to 482 μs without changing the argmax. On the RTX PRO 6000 that certified head is slower (2598 vs. 1750 μs), so the LM head uses W12 there.
Qwen3.8-27B runs Gated DeltaNet on 48 layers, each keeping a fixed-size recurrent state. A snapshot per tree node would be about 18 GB at 24 nodes, so verification recomputes state along each root path. A WY kernel turns the rank-one updates into matrix products and takes 48 layers from 1485 μs to 542 μs. Edge scores are the selector log-probabilities given the parent. The drafter runs 16 positions and the tree verifies 32 nodes. DSpark's Markov head also conditions on the previous token, so the same builder applies.
Two-level verification reads the 8-bit view already nested in each W12 record. A Q pass marks a path and does not touch state. The drafter then drafts again from the end of that path. Only the following exact pass accepts tokens. The two passes share a top-1 token at 98.7% of positions.
Batching only chooses rows. Requests share a node budget, 192 nodes at 24 FP8 requests, on a flat stretch of the target GEMM's cost. Each request drafts 16, 12, or 8 positions from recent acceptance. A new prompt joins that same weight read only when the scheduler would otherwise hold it.
Four targets, three drafters, two GPUs, five benchmarks, greedy decoding with thinking on. Figures are speedup over autoregressive decoding and τ, as an equal-weight geometric mean.
| Setting | stock | SpecPilot |
| 1 request, H100, Qwen3.8-27B + DFlash2 | 3.95× / 4.65 | 8.86× / 13.08 |
| 1 request, H100, same model + DSpark | 3.74× / 4.42 | 9.41× / 12.85 |
| 1 request, RTX, same model + DFlash2 | 4.46× / 5.23 | 10.87× / 12.80 |
| 1 request, RTX, Muse-Glimmer-30B | 4.69× / 5.16 | 10.63× / 12.64 |
| Batched H100, FP8 + DSpark | 1.94× / 4.37 | 3.93× / 5.42 |
| Batched H100, Qwen3.5-4B | 1.51× / 6.58 | 3.77× / 6.97 |
| Batched H100, Gemma-4-12B + EAGLE-3 | 0.68× / 2.04 | 2.09× / 2.54 |
The abstract's 4.46× to 10.87× is the RTX Qwen3.8-27B cell. Under two-level verification, τ is tokens per exact pass, and two drafts share that pass, which is why τ moves from a bit over 4 to about 13. A tree alone, in replay, lifts the 8-position chain from 4.25 to 5.38. On H100 with one request, speed is 1.84× the DDTree baseline, and DDTree itself is 4.82× over autoregressive decoding. Sealed gains on unseen requests are 2.13–2.44× over stock for one request and 1.85–3.11× batched. FP8 DFlash2 on H100 goes from 2.02× to 3.90×. At BF16 with 6 to 10 requests, speedup over autoregressive decoding goes from stock's 3.20× to 5.67×. On Gemma, stock is slower than autoregressive decoding.
The H100 single-request curve moves from 184 tok/s to 416. Trees reach 270, the next round cuts weight bytes, and two-level verification then drops the one-exact-pass step. In that same setting, a single agent without the cost model finishes three rounds at 1.34× stock. SpecPilot finishes at 2.13×. Weight reads were 80% of the step, and the single-agent loop never touched them.
Without speculation, W12 is 1.19–1.28× BF16 autoregressive decoding at one request on Muse-Glimmer-30B, Qwen3.8-27B, Gemma-4-12B, and Llama-3.1-8B. Weights shrink 14–22%, and the KV pool on Qwen3.8-27B grows 2.0×. From 1 to 32 rows the GEMM is 1.17–1.47× cuBLAS. Stock was already tuned: on the RTX, block length 12 plus Triton attention is 9.9% faster than the H100 configuration copied across.
The leftover speedup is in the deployment. One request is bandwidth-bound, so wider checks, a cheap pre-pass, and thinner weights pay. Batching is per-row compute, and those same changes lose unless the server spends rows only where they pay. A SGLang deployment of DFlash, DSpark, or EAGLE can keep its weights and its greedy outputs and still reach a bit over 2× stock at one request. The RTX PRO 6000 has no wgmma. Porting the H100 tree and W12 is a sealed 1.92× over stock there, and two-level verification brings it to 2.44×.
Lossless means greedy decoding, plus near-ties from floating-point reduction order. At the first divergence, stock and autoregressive log-probabilities differ by at most 0.232 nats. Sampling is unmeasured.
Speedups over autoregressive decoding come from interleaved runs on development requests, within −2% to +6% of the sealed gains on unseen requests. That gap is large enough to matter for an SLA. The search used Claude Opus 5.5. GPU hours, the share of failed directions, and how much of the 140 entries was finished by hand are unreported.
The sign of a trick flips with the regime. A 24-node tree loses 14.9% on FP8 batching, and two-level verification loses 13% on BF16 batching. The Q pass already costs 0.80–0.88 of a step, so a cycle needs nearly twice the tokens to break even. The byte floor's 60% of peak compute and 90% of peak bandwidth are estimates. The RTX system beat a 2.12× ceiling because that ceiling assumes one draft per exact pass. Changes that redefine the step are invisible to the cost model up front. Token forcing is calibrated to stock's existing 0.72% mismatch. A bug that drifts further, but stays inside 0.5 points, still passes.