Speculative decoding: the trick behind Google's 2-3x faster LLM inference in production
hongyangzh · x · 2026-09-13
A deep-dive article explains speculative decoding — used in production by Google (AI Overviews in Search), Anthropic, and Meta — which speeds up LLM inference 2-3x. A cheap draft model proposes several tokens at once, and the large target model verifies the whole block in a single forward pass, since low-batch autoregressive decoding is memory-bandwidth bound and leaves most GPU compute idle. The article covers internals, code, tradeoffs, and what's replacing the two-model setup.
More from Infra
- xAI's Memphis datacenter now has its own power sources en route to $100b ARR — PaulYacoubian · 2026-09-13
- DeepSeek's V4.1-Flash KV cache compression could undercut OpenAI and Anthropic's compute moat — justlikemedics · 2026-09-13
- Google to spend $15.1B on Finnish AI infrastructure, including three data centers and 22-year nuclear deal — Beth_Kindig · 2026-09-13
- Dev picks LFM2.5-2.6B and MiniCPM5-2B as favorite edge-device models — reach_vb · 2026-09-13
- Prompt caching can inflate your LLM bill: check write premiums, TTL and actual reuse — gethackteam · 2026-09-13
- TRL v1.13 ships long-context training: 1M+ token sequences on a single 8-GPU H100 node — SergioPaniego · 2026-09-13