Evolution of speculative decoding: from parallel guessing to sequential drafters
LysandreJik · x · 2026-11-12
A thread comparing two foundational speculative decoding papers:
- Blockwise Parallel Decoding (the first speculative decoding paper) produces all speculated tokens at once, independently — fast, but token N+2 can't see N+1, hurting accuracy.
- Fast Inference from Transformers via Speculative Decoding adds a small drafter model beside the main model: for each main-model forward pass, the drafter runs k sequential passes producing k tokens. Since token N+2 sees N+1, guess quality improves and more speculated tokens are kept — overall generation is still faster despite the extra small passes.
More from Infra
- Compute is going fungible: Anthropic roadshow backs Ben Bajarin's hyperscaler thesis — BenBajarin · 2026-11-13
- Liquid AI's 74M-param LiquidCodec hits 361x realtime encoding on a single CPU thread — helloiamleonie · 2026-11-13
- AI Agent Traffic on Cloudflare Up 1,700% in a Year as Over Half of Web Traffic Turns Non-Human — PeterDiamandis · 2026-11-13
- One 8×B200 node serves 3,000 DeepSeek agents, 19x official capacity — james_y_zou · 2026-11-13
- Outlook is down for thousands of users — and Microsoft's status page is down too — dkpk · 2026-11-13
- Tracking data claims 1.5GW of restricted Nvidia chips still ended up in China via smuggling — zephyr_z9 · 2026-11-13