DFlash 2 parallel drafting lifts speculative decoding acceptance 21% for 1.3% latency cost
songhan_mit · x · 2026-11-06
Zhijian Liu, co-founder of incoai, shared a key insight from his #Runtime talk: drafting one token at a time becomes its own bottleneck in speculative decoding.
Method and results:
- DFlash drafts a whole block of tokens in parallel, making room for a deeper, more accurate drafter while keeping latency low
- The target model still verifies every token, preserving correctness
- Built on this, DFlash 2 accepts about one more token per verification pass (+21%) for just 1.3% more cycle latency
The work targets faster inference for AI applications and agents.
More from Infra
- Jane Street shares its experience building with NVIDIA's cuTile language — GuggerSylvain · 2026-11-06
- AI overbuild skeptics miss the demand: agents, cyber AI and 0.1% enterprise penetration — builderjaydub · 2026-11-06
- Against datacenter doom: three reasons the buildout continues — max_paperclips · 2026-11-06
- iQuantize: open-source desktop workspace for guided LLM quantization to GGUF, MLX and ONNX — Zipidyzip · 2026-11-06
- Developer ships server-side barge-in for local LLM voice assistant VoiceManager — HockeyDadNinja · 2026-11-06
- LSEG cuts 5-year market backtest from hours to ~1 minute; Tampa General adds AI safety gate — NVIDIAAI · 2026-11-06