Thread: speculative decoding speeds up inference without changing weights or sampling
cneuralnetwork · x · 2026-10-22
A thread explains the elegance of speculative decoding: it doesn't make the target model smarter or change its weights—it exploits the gap between sequential generation and parallel verification. The speedup depends on the draft model, acceptance rate, KV cache management, batch size, and verification overhead. Key formula: the expected acceptance probability at a prefix is α = Σₓ min(p(x), q(x)) = 1 − TV(p, q), so a draft distribution closer to the target accepts longer prefixes and yields more tokens per verification pass. Faster inference while preserving the exact sampling distribution is a beautiful systems optimization.
More from Infra
- H100 Rental Rates Keep Climbing Across the Entire Term Structure — isidentical · 2026-10-22
- Phonon TTS and Fluid Speech Mini: a day-long test of lightweight local speech models — gregbarbosa · 2026-10-22
- Databricks AI Runtime goes GA: GPUs in seconds, adopted by Robinhood, Superhuman, Merck — matei_zaharia · 2026-10-22
- Flint MoE model lands in llama.cpp: 39.8B total params, only ~2.5B active per token — jacek2023 · 2026-10-22
- Omarchy package server is timing out: updates failing on cursor, grok-bot, codex-desktop packages — mark_k · 2026-10-22
- AWS shows how to cut agent costs with local SLMs routing to Bedrock only when needed — AWS ML Blog · 2026-10-22