Draft model hits ~60 tok/s running Qwen3.8-27B at 131k context on a 16GB GPU
pneuny · reddit · 2026-09-12
- Using the DFlash2 draft model (HermiHg/Qwen3.8-27B-DFlash2-Q2KS-MIX-GGUF) with an IQ3XXS main model at 131k context, the author averaged 60 tokens/s on a 16GB RX 9070 XT.
- It beats the built-in MTP speculative decoding, which multiplies VRAM requirements; for single-thread mode this is among the best options for a 16GB GPU.
- Caveat: speculative decoding is more sensitive to VRAM overflow — if you exceed available VRAM, turning it off is actually faster on his DDR5 PCIe 5 system.
- More extensive testing on different cards is documented in the HF discussion thread.
More from Infra
- TSMC reportedly raising CoWoS packaging prices up to 20% next year on AI chip demand — Beth_Kindig · 2026-10-31
- effect-harness on Cloudflare: run agent loops, tools and credentials on separate VMs — samgoodwin89 · 2026-10-31
- Google's Project Suncatcher boots TPU in orbit and beams back first AI response — sundarpichai · 2026-10-31
- Together AI launches Post-Training platform for agentic RL with tool-use environments — togethercompute · 2026-10-31
- General Compute raises $100M for inference capacity as CoreWeave stock hits 12-month low — YvesMulkers · 2026-10-31
- Analyst warns of unstable AI compute feedback loop as Kioxia's NAND price warning triggers flash crash — bookwormengr · 2026-10-31