Qwen 27B with vision on a 16GB GPU: 85K context at 45 tok/s, full config
FerLuisxd · reddit · 2026-09-10
A detailed local-deployment recipe for running Qwen 3.8 27B with vision on a 5060 Ti (16GB VRAM): IQ3XXS-mtp GGUF quant from ISTA-DASLab, BF16 mmproj, via beellama.cpp. Result: 45 tok/s decode, 300 tok/s prefill, 85K context, with 1.5GB headroom left.
Key tricks: full GPU offload with direct-io, kvarn4 KV-cache compression for both K and V (author links benchmarks showing acceptable quality loss), draft-MTP speculative decoding (max 2 draft tokens), and the option to move mmproj to CPU to free more VRAM. Full config block included; author invites other 16GB sweet-spot configs.
More from Infra
- $5M Seed Round Can't Afford One DeepSeek Training Run: GPU-less AI Startup's Three Paths — FinanceYF5 · 2026-10-26
- HBM demand shifts: low-stack inference plans shelved, Rubin to go 12-high only — BenBajarin · 2026-10-26
- CircuitHub unveils The Grid, a factory-scale robot making PCBs in the US twice as fast as Shenzhen — garrytan · 2026-10-26
- Ubicloud publishes "Fixing Tool-Call Accuracy in vLLM" — petrusenko_max · 2026-10-26
- Meta approves nonvoting Class C shares as 2026 capex guidance hits $135B–$145B — thione · 2026-10-26
- Unsloth Ships MiniMax-M3.1-Flash Quants: 2-bit Retains 91.5% Accuracy, Runs in 150GB RAM — danielhanchen · 2026-10-26