Acceptance is not speed: relaxed speculative decoding inflates output length by 1.54x
hongyangzh · x · 2026-10-24
An arXiv paper shows that training-free lossy speculative decoding accepts 1.34x more draft tokens per round across 12 model pairs, but produces 1.54x longer completions — mostly inflated reasoning traces — canceling most gains. Rules cutting verifier rounds also drop accuracy. Acceptance count alone is a misleading speed metric.
More from Research
- Alignment eval canary strings dismissed as more talisman than safeguard — tszzl · 2026-10-24
- Are we solving LLM memory by remembering more when the real problem is knowing what to forget? — Prestigious-Tip6030 · 2026-10-24
- Victor Taelin: formal verification explodes as AI collapses proof costs — pwlot · 2026-10-24
- DeepSeek-AI and Peking University publish calculus for spatiotemporally composable, self-evolving agent software — akbirthko · 2026-10-24
- SCOPE compiles natural-language policies into neurosymbolic programs to constrain coding agents — 1982_miguel · 2026-10-24
- Stephen Wolfram proposes 'Ruliobiology': a computational theory of biological form — pickover · 2026-10-24