Running Qwen 3.8 27B at 75t/s on 16GB VRAM: A Practical Guide
Kernoriordan · reddit · 2026-08-31
The author shares their experience tuning Qwen 3.8 27B on an RTX 4080/5080 (16GB VRAM), achieving an average decode speed of 75t/s (peaks at 100t/s) using a specific quantization and llama.cpp configuration.
Key Setup:
- Model: Qwen3.8-27B-i1-IQ4XS-GGUF-Smaller (Hybrid quantization designed for 16GB VRAM and Multi-Token Prediction)
- Critical Args:
- -ngl 99: Offload all layers to GPU
- -c 85000: Context length
- --spec-type draft-mtp: Enable Multi-Token Prediction speculation
- --ctk q40 / --ctv q40: Quantize KV Cache
Logs demonstrate consistent performance during long text generation.
More from Infra
- RL train-sample systems are frontier AI's hardest infra problem — and a trade secret — brianryhuang · 2026-10-16
- Local tuning of GLM-5.3-Flash on 2×GB10 lifts decode from 49 to 58 tok/s — EAccelerate_42 · 2026-10-16
- 3x DGX Spark cluster runs GLM-5.3-Flash locally behind a proxy, 1.75M tokens/day at 119W — Teknium · 2026-10-16
- Optuna creator Akiba: when 2 hyperparameters cost $20M, what's left for BO/HPO — Stefania_druga · 2026-10-16
- Amazon Proposes Per-Request Validated Evidence Caching for Production RAG — _reachsumit · 2026-10-16
- LinkedIn's TransX report: sequence transduction for recsys, +6.02% CTR in A/B tests — _reachsumit · 2026-10-16