DFlash2 Speculative Decoding: Qwen3.8-27B Hits 86.7 tok/s on 4080 16GB
Apprehensive_Bar6609 · reddit · 2026-08-26
A detailed technical guide demonstrates accelerating Qwen3.8-27B on an RTX 4080 16GB using DFlash2 speculative decoding.
- Mechanism: DFlash2 uses a smaller Drafter model to predict tokens in parallel, verified by the larger Target model in a single pass, achieving lossless speedups.
- Setup Steps:
- Build llama.cpp from the unmerged PR #27342 branch.
- Download the target model (Qwen3.8-27B GGUF) and its specific Drafter model (561MB).
- Launch llama-server with specific flags, notably --spec-type draft-dflash and --model-draft.
- Performance Data:
- 4k context: Throughput increased from 50.6 to 86.7 tok/s (1.72x speedup).
- 120k context: Throughput reached 66.1 tok/s (1.31x speedup).
- VRAM cost: Only adds 561MB.
- Caveats: Drafter models are pair-specific; nmax=3 offers the best tradeoff; a PR bug limits stable context to 122k tokens.
More from Infra
- Cerebras vows multi-generation wafer-scale roadmap to keep fastest-inference crown — Sethwinterroth · 2026-08-26
- NVIDIA Groq 3 LPX Enters Production for Agentic AI Speed — badumtsssst · 2026-08-26
- Expert doubts claim that data centers use less water than rainfall — tdietterich · 2026-08-26
- Apple M7 criticized for weak AI readiness vs CUDA, Agents shift away from consumer hardware — teortaxesTex · 2026-08-26
- Toloka Train Cuts AI Costs Up to 37x with Fine-Tuning and Gisting — MParakhin · 2026-08-26
- Flaw in anti-finetuning: Cost > Quality once models are saturated — rhythmrg · 2026-08-26