DFlash2 Speculative Decoding: Qwen3.8-27B Hits 86.7 tok/s on 4080 16GB
Apprehensive_Bar6609 · reddit · 2026-08-26
A detailed technical guide demonstrates accelerating Qwen3.8-27B on an RTX 4080 16GB using DFlash2 speculative decoding.
- Mechanism: DFlash2 uses a smaller Drafter model to predict tokens in parallel, verified by the larger Target model in a single pass, achieving lossless speedups.
- Setup Steps:
- Build llama.cpp from the unmerged PR #27342 branch.
- Download the target model (Qwen3.8-27B GGUF) and its specific Drafter model (561MB).
- Launch llama-server with specific flags, notably --spec-type draft-dflash and --model-draft.
- Performance Data:
- 4k context: Throughput increased from 50.6 to 86.7 tok/s (1.72x speedup).
- 120k context: Throughput reached 66.1 tok/s (1.31x speedup).
- VRAM cost: Only adds 561MB.
- Caveats: Drafter models are pair-specific; nmax=3 offers the best tradeoff; a PR bug limits stable context to 122k tokens.
More from Infra
- MiniCPM-V 4.7 runs a 35B MoE multimodal model on phones with just 3GB memory — nevrekaraishwa2 · 2026-10-15
- Nvidia Is Now Worth More Than the Entire Indian Stock Market — theteknosaur · 2026-10-15
- NVIDIA joins EU's AI in Science Enablers, offering compute, training and cloud resources — nordicinst · 2026-10-15
- Open source's hardware trap: data centers crowd out consumer GPUs, local LLMs become a privilege — Dante_77A · 2026-10-15
- NVIDIA backs EU AI in Science push as 35 supercomputers to serve 3M researchers — NVIDIA Blog · 2026-10-15
- Jensen Huang: 2027 AI bottleneck is 35GW of power, revenue justifies the capex — BG2 Pod · 2026-10-15