Speculative decoding: the trick behind Google's 2-3x faster LLM inference in production
hongyangzh · x · 2026-09-13
A deep-dive article explains speculative decoding — used in production by Google (AI Overviews in Search), Anthropic, and Meta — which speeds up LLM inference 2-3x. A cheap draft model proposes several tokens at once, and the large target model verifies the whole block in a single forward pass, since low-batch autoregressive decoding is memory-bandwidth bound and leaves most GPU compute idle. The article covers internals, code, tradeoffs, and what's replacing the two-model setup.
More from Infra
- Would LITE accepting MRVL's merger offer prove its moat was overstated? — BenBajarin · 2026-10-30
- Claymation body horror on 8GB: three wrong settings, not the GPU, made local renders slow — roro9- · 2026-10-30
- durable-object-migrator: zero-downtime exports for Cloudflare Durable Objects — samgoodwin89 · 2026-10-30
- An unnamed firm already runs a fairly scaled dedicated vehicle for compute lending, says analyst — matt_slotnick · 2026-10-30
- GPU break-even throughput vs open-weight token prices up 25-83% since May — bookwormengr · 2026-10-30
- Sam Altman says the chip supply chain's invisible hand is failing — and DRAM is why — SnoozeDoggyDog · 2026-10-30