Speculative decoding explains why coding agents type fast on boilerplate but slow on new logic

RunAI_Coder · reddit · 2026-09-09

The author kept blaming "slow afternoons" on the model until reading AMD and Embedded LLM's speculative decoding write-up, which explains the behavior precisely.

How speculative decoding performs

A small draft model guesses the next few tokens; the big model verifies them in one pass. Outputs are unchanged—only tokens per expensive pass increase.
On gemma-4-26B-A4B-it with Google's paired draft: first drafted token accepted 94% of the time on GSM8K, fifth 66%. On MBPP (Python): 89% and 49%.
Code acceptance drops because "formatting, identifiers, and implementation choices can cause an otherwise plausible continuation to diverge."
Best config hits 2.87x plain decoding; Qwen3-8B with an EAGLE-3 draft ran at 0.44x–0.88x—slower than no speculation. Drafting costs even when guesses land.

Other clocks in a turn

Before the first token: the whole context is re-read—agent transcript plus every pasted file. A cache miss after compaction, an edited system prompt, a changed tool list, or a cache TTL pause forces a full re-read.
During typing: load matters. Both major vendors now price a 2.5x faster output lane, implying the standard lane is the slow one.
Many of the longest waits contain no model at all—they're the test suite.

The author asks whether anyone has logged per-turn TTFT vs decode time vs tool time on a hosted API across a full day, to separate load from one's own cache misses.

Original post →

Speculative decoding explains why coding agents type fast on boilerplate but slow on new logic

More from coding & agent