Custom UD-IQ4_XS quant runs Qwen3.8-27B with MTP at 68 tok/s on a 16 GB card
ZestRocket · reddit · 2026-10-10
A Redditor wanted the uncensored Qwen3.8-27B (llmfan46's Heretic build) with MTP speculative decoding on a 16 GB RTX 4080, but no existing quant fit both, so they replicated Unsloth's per-tensor UD recipe and shipped 12/16/24 GB versions, benchmarking KLD against Q80 for every public quant.
Key results
- Quality: UD-Q5KXL hits 0.0045 mean KLD; UD-IQ4XS (16 GB) lands at 0.0268 prose / 0.0192 code — a huge jump over similarly sized 3-bit quants (0.06+).
- Speed with MTP on: 55 tok/s code, 50 tok/s prose vs 29/29 without.
Surprises
- 2 draft tokens beat 3 (68 vs 56 tok/s) due to fewer rejections.
- 40K vs 48K context produced identical tokens while being 22% slower — pure VRAM spill into shared memory.
- The 16 GB edge is brutal: desktop apps holding 1.9 GB vs 1.2 GB of VRAM dropped code gen from 68 to 42 tok/s; opening Chrome mid-generation crashed it to 0.2 tok/s.
- MTP head precision barely matters (83% acceptance at q6K vs IQ3S).
The repo includes all three GGUFs, commands, and scripts for recipe extraction, dry-run verification, KLD measurement, and seeded speed benchmarks.
More from Infra
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Jensen Huang lays out NVIDIA's full Vera Rubin agentic system: NVL72, MGX, CMX and RTX — rohanpaul_ai · 2026-10-23
- ONNX to Transformers.js: running models directly in your browser — JosephJacks_ · 2026-10-23
- MiniMax M3.1 Flash free for 3 days on 0G, with 1M context and 512K output — MiniMax_AI · 2026-10-23
- "We bet on Cloudflare and it paid off": R2 and D1 at billion scale — dinasaur_404 · 2026-10-23
- Public data is drying up; private training data is turning into a browsable catalog — SucceededMind · 2026-10-23