AgSpec: per-agent draft lengths and diff-format indexing lift coding-agent decoding to 4.76x

AgSpec: Pushing the Limits of Retrieval-Based Speculative Decoding in Coding Agent Pipelines

Sumin Lee, Sukmin Cho, Suengjae Lim, Youngjin Kwon

cs.CL

2026-10-01

KAIST's AgSpec indexes session, workspace, and global text in the format agents emit and adapts draft length per agent, lifting coding-agent decoding throughput up to 4.76x over AR decoding.

What problem this solves

Coding agents have an unusual latency profile. One user request becomes a multi-turn session; specialized roles such as a localizer, coder, and executor alternate between model calls and tool runs, failed attempts trigger retries, and each turn regenerates much of what earlier turns produced: code, test logs, previous patches. Speculative decoding (SD) should be a natural fit: a drafter proposes several tokens, the target model verifies them in one parallel pass, and every accepted token saves a sequential step without changing the output distribution. Retrieval-based SD skips the draft model entirely by copying continuations from text the system already holds, which suits code, where identifiers and conventions repeat heavily.

Existing retrieval methods capture only a fraction of this in agent pipelines. A draft helps only if the reused text is present in the corpus and stored in the form the agent emits. Current corpora cover either the current request or a static datastore; a file truncated out of the prompt becomes unreachable, and a retry cannot see the previous attempt once its call ends. Format is the subtler gap: a coder reads files as they stand but writes unified diffs, where every copied line gains a prefix character the file never contained, so an index built from raw files cannot match the emitted token sequence. Draft length is equally coarse: fixed caps or match-scaled factors, even though measured accept length sits in a distinct range per agent and drifts across turns.

Method

AgSpec is a framework layered over an existing retrieval engine, supporting the suffix automaton of SAM-Decoding and the suffix tree of SuffixDecoding; the engine itself is untouched.

Corpora are split by source. The session corpus keeps prompts, tool outputs, and generated tokens for the whole session, shared across calls and agents, so earlier attempts stay retrievable. The workspace corpus indexes repository artifacts the session opens, storing each file in the agent's emission format as well: under unified diffs, one copy with every line prefixed by a space and another with a minus, so any line the coder copies into a patch matches. The global corpus is built before serving from agent trajectories on disjoint tasks (70 SWE-bench tasks, 213 TeamBench tasks) and stays fixed. Retrieval queries all three with the same suffix; between the live corpora the longer match wins with ties going to the session, and the global corpus is used only when it beats the live match by δ = 5 tokens, since a long match in unrelated text is usually coincidence.

Draft length is K = clip(round(α·m), 1, Kmax), with m the match length. Kmax is set offline: replaying past trajectories gives the acceptance probability of each agent's k-th draft token, and the cap keeps positions down to the last one whose expected gain still covers its verification cost. α adapts online from verification feedback, starting at 1.0, growing when a draft is fully accepted and shrinking when it is cut off. The update is a stochastic subgradient step on the pinball loss, so α tracks the τ-quantile of the accept-to-match ratio as it drifts within a session, and scales persist across sessions.

Results

Main comparison on SWE-bench Verified and TeamBench with Devstral-24B, Gemma3-27B, and Qwen3.6-27B (all 4-bit quantized) at batch sizes 1 and 16, against AR decoding, PLD, REST, FastCoder, SuffixDecoding, SAM-Decoding, and EAGLE-3.

SettingAgSpecReference
Both benchmarks, 3 models, b=12.27–4.37× over ARhighest or second in every setting
Same, b=16up to 4.76× over AR18.0% above the fastest prior on average
Gemma3-27B, SWE-bench, b=163.61×SAM-Decoding alone: 2.72×
LiveCodeBench (no repository)4.87–7.94× over AR35–145% above the fastest prior
Terminal-Bench (single agent)9.3–27.2% above the fastest prioraccept length +16.7–34.3% vs the engine baseline

Layering AgSpec on the engines alone: SuffixDecoding gains 16.5% throughput on average and SAM-Decoding 64.6%, with its accept length nearly doubling. Vanilla SAM-Decoding lost to SuffixDecoding in all twelve settings; with the framework it wins nine.

Ablations separate the sources. Degrading the session corpus to per-call storage costs 39–51% of accept length, and the full session corpus is a 1.6–2.1× gain over per-call, the largest single factor. The workspace corpus adds 26–47% alone and 2.2–7.0% on top of the session; all three layers reach 2.9–4.8× the accept length of the global corpus alone. Diff-format indexing mainly rescues the first turn (coder AL +13–39%); by turn five the gain falls under 3%, because the agent's own patches have entered the session corpus. On draft length, a fixed K=32 is 14% faster than K=8 at batch 1 but 30% slower at batch 16, while AgSpec is fastest at both; the online scale alone, on fixed K=32 at batch 16, lifts throughput 66% and cuts the rejected-token share from 78% to 50%. Overhead stays small: corpus queries and draft selection take 1.1–5.5% of each step, and the global corpus sits in 2.5–4.3 GB of host memory.

Why it matters

For anyone serving coding agents this is directly usable: implemented in vLLM, training-free, no engine modification, and verification preserves the target model's output distribution. The ablations carry a plainer lesson: simply retaining the session trajectory for retrieval captures most of the gain, and the workspace and emission-format layers cover what the session has not accumulated yet. It also holds beyond the headline setting, on a repository-free benchmark and a single-agent pipeline. The honest framing is that it accelerates decoding only; task accuracy does not move, and the ceiling depends on how much reusable text the workload contains.

Limitations

The authors note that letting the workspace corpus grow across sessions brings no meaningful benefit, leaving multi-session shared repositories to future work, and that α does not converge but fluctuates around a moving target. Beyond that: on SWE-bench, the break-even λ used to set caps was measured with Devstral and reused for all three models; all experiments run on a single GPU with greedy decoding and 4-bit quantized models, so sampling and larger batches are untested; the global corpus requires trajectories from similar tasks, and cross-domain transfer is not evaluated; and session and workspace corpora contain prompts, repository content, and tool outputs, where the ethics statement only asks deployments to isolate them per session and discard them at session end.

Terms

Source

Related papers

All paper explainers