Qwen 27B Q4 with 100K context at ~30 t/s on a 16GB AMD RX 7800 XT: full guide
Haunting-Stretch8069 · reddit · 2026-09-29
The author runs Qwen 3.8 27B Q4 on a 16GB AMD RX 7800 XT at 30 t/s decode with 100K context, showing it's feasible on 16GB VRAM.
Key steps:
- Build llama.cpp with Vulkan instead of ROCm (ROCm uses more VRAM and leaks);
- Use unsloth's Qwen3.8-27B-UD-IQ4XS.gguf plus mmproj-F16 for vision;
- Quantize KV cache to q80 (K) and q51 (V) to save memory;
- Enable flash-attn, disable unified KV, set 100096 ctx with checkpointing;
- Full copy-paste llama-server command with sampling params included.
More from Infra
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16
- Stratechery's Fictional Obituary: Imagining Jensen Huang's 2026 Exit From Nvidia — Gregory_C_Allen · 2026-11-16
- vLLM serves DeepSeek V4.1-Flash at 954K tokens/s per GPU, $38.96/GPU-hour on GB300 NVL72 — bookwormengr · 2026-11-16
- Colossus 2's 1.2M B200s could generate ~$400B/year in tokens at DeepSeek Flash rates — bookwormengr · 2026-11-16
- BIS issues rule blocking remote access to US AI chips via third-country data centers — teortaxesTex · 2026-11-16
- Qdrant 1.20.1: 50% fewer cold-storage reads, ~2x faster quantization, agent-ready setup — qdrant_engine · 2026-11-16