Guide: squeeze ~18-20 tok/s from Qwen3.8-27B on 16GB VRAM with lossless KV cache

BassAzayda · reddit · 2026-08-18

The author pushed Qwen3.8-27B from 8–12 tok/s to 18–20 tok/s on an RTX 5070 Ti (16GB VRAM) + 64GB system RAM while keeping a lossless q80 KV cache at 65k context. The key move: drop plain -ngl layer offloading and use --override-tensor to offload only FFN weight matrices (1 of every 3 layers) to CPU, freeing 2.8GB VRAM so 100% of attention operators stay on GPU.

Other tricks:

MTP speculative decoding (--spec-type draft-mtp, draft length 2) for 30–50% throughput gain;
Pin generation threads (-t 8) to physical performance cores, let prefill batch threads (-tb 16) use hyperthreads;
Set GGMLCUDADISABLEGRAPHS=1 on RTX 50-series/Blackwell to avoid CUDA graph capture stalls.

A full copy-pasteable llama-server launch script is included (IQ4XS quant, 65,536 context, q80 KV cache, prefix caching, etc.).

Original post →

Guide: squeeze ~18-20 tok/s from Qwen3.8-27B on 16GB VRAM with lossless KV cache

More from Infra