vLLM Speculators v0.9.0: 2x faster speculative decoding training, 50% less storage
vllm_project · x · 2026-10-20
vLLM project shipped Speculators v0.9.0 with major training and serving optimizations for speculative decoding:
- A fused loss kernel speeds up forward training passes up to 2x on 1B models
- An FP8 hidden-state backend cuts storage 50% with no quality loss
- Mooncake connector improvements: transfer manifests returned via kvtransferparams for simpler tensor packing
- torch.profiler integration for GPU kernel breakdowns, plus a throughput-vs-interactivity benchmark utility
- Adds GLM 5.3 hidden-state extraction and DFlash 2 strict candidate selection; Python 3.10 deprecated
Trained speculators can be served directly in vLLM.
More from Infra
- Shadeform launches six features as agent-era compute breaks existing rental models — omarsar0 · 2026-10-20
- Alex Atallah's CAP Theorem for Inference Providers: Reliability, Cost, Speed — AccBalanced · 2026-10-20
- Maxwell: a model that rewrites running binaries to speed up Whisper and LTX-2, with formal verification — EvilHackFar · 2026-10-20
- New paper shows whitening-based inner optimizers enable strong replica scaling in DiLoCo LLM pretraining — kastnerkyle · 2026-10-20
- Together Launches Post-Training Service for Custom Models via RL and SFT — togethercompute · 2026-10-20
- Gemini 3.8 Flash missing in Mumbai region, devs weigh Google's data residency churn — Tech_Partner_01 · 2026-10-20