ASD: training-free approximate speculative decoding boosts LLM throughput up to 15.26%
新智元 · wechat · 2026-09-05
A joint team from Beihang, Tsinghua, HKU and Peking University proposes Approximate Speculative Decoding (ASD), which relaxes the 'stop at first token mismatch' rule: it selectively accepts low-regret mismatches within a strict request-level budget and reuses the already-computed greedy suffix behind them.
Results:
- +7.78% average throughput on Qwen3-14B + DSpark-14B across 7 tasks (up to +11.73%);
- Positive gains in all 10 setups across DSpark/EAGLE3/Medusa drafters (3.05%–15.26%, avg 7.52%);
- +10–16% verifier acceptance rate on the 284B DeepSeek-V4-Flash (8x H20), with accuracy drift under 0.30pt;
- Plugin verifier costs only 0.045–0.083ms per token, no training/fine-tuning/extra forward passes; budget=0 falls back to strict decoding.
The paper is explicit that accepting non-greedy tokens changes the trajectory — not lossless — but audits show accuracy holds on most tasks, arguing speed/behavior tradeoffs should be explicitly modeled and auditable.
More from Infra
- The best local model you can run on 2 GB10s, per this desk setup — jasonkneen · 2026-09-06
- One ComfyUI node fixed MiniMax H3 OOM on RTX 5090: full workflow for 15s 2K video — denizbuyukayak · 2026-09-06
- KV cache often spills out of HBM in the agentic era, tanking effective bandwidth — AccBalanced · 2026-09-06
- Hybrid bonded HBM hypothetical market: over 3 billion D2D applications per year — zephyr_z9 · 2026-09-06
- Ollama CEO: open models will carry 80-90% of enterprise tokens at just 10-20% of cost — victor_explore · 2026-09-06
- Nvidia de-specced Rubin Ultra HBM from 12-Hi to 8-Hi: $/bandwidth is the bottleneck — AccBalanced · 2026-09-06