Tutorial: speculative decoding in vLLM to cut LLM latency and double tokens per second
MaiaStudios · reddit · 2026-09-20
A hands-on Python tutorial showing how to implement speculative decoding in vLLM to reduce LLM inference latency.
Key claim: with speculative decoding you can roughly double tokens generated per second. Useful for anyone self-hosting models on vLLM and looking to cut latency.
More from Infra
- Qwen3.8-27B on a 7900XTX hits 40 tok/s with 240K context for local agentic coding — W61k3r · 2026-09-20
- "There are more inference workloads in Heaven and Earth, Horatio" — a quip on overfit optimization — charles_irl · 2026-09-20
- Kimi subscriptions return after roughly two months, suggesting Moonshot found more compute — ChrisGPT · 2026-09-20
- HN: How OpenAI Used Its Own LLMs to Design Its Jalapeño Chip — petrusenko_max · 2026-09-20
- FlashNorm: two lines of algebra buy 33-35% speedup — and a CUDA race that made the model echo the past — AI Engineer · 2026-09-20
- NEAR AI Brings Confidential Inference to Bittensor Subnet SayGm, an OpenRouter-Style Router — markjeffrey · 2026-09-20