Qwen3.8-27B on Free 2x T4s: DFlash2 Hits 154 tok/s on Code
Ishannaik · reddit · 2026-10-25
A hands-on attempt to squeeze inference out of Qwen3.8-27B on Kaggle's free 2x T4s (no P2P, 70W cap per card): EXL3 4.0bpw with exllamav3, tensor parallel over PCIe, and a DFlash2 block drafter (7 tokens) with greedy + exact matching vs a no-draft baseline.
Results over a 38-prompt suite: overall 55-57 tok/s (vs 18-19 without drafting, 3x); HumanEval at 154 tok/s; prose at 35-37 tok/s. Same-machine comparisons: llama.cpp best was 16.5 tok/s (MTP crashed on tensor split); vLLM int4 + its own MTP drafter reached 40.8 tok/s.
The wall is prose acceptance: 2.5 tokens accepted per step vs 7 on code. The author asks the community how to push prose acceptance above 2.5, whether anyone has tried switching drafters mid-generation (short block for prose, long for code), and tricks for TP all-reduce on PCIe cards without P2P. Notebook shared on request.
More from Infra
- Izwi 0.1.0 ships fully local voice AI with TTS, cloning and diarization — zinyando · 2026-10-26
- Frontier security models will need new compute classes, quantum likely first — BenBajarin · 2026-10-26
- DesignArena data: Gemini 3.8 Flash quality degrades sharply weeks after launch — legit_api · 2026-10-26
- Perplexity's 5B decision model runs free on your Mac, distilled from its 7B model — Saboo_Shubham_ · 2026-10-26
- Chart: Data centers funded via SPV structures keep hyperscaler debt off balance sheet — SumitGup · 2026-10-26
- OpenAI built an 'offline world' of cached web content to train frontier RL runs like GPT-7 — daniel_mac8 · 2026-10-26