RTX 3090 Qwen3.8-27B deployment: vLLM outperforms llama.cpp
Lower-Ad6101 · reddit · 2026-08-28
A Reddit user shared a detailed optimization guide for deploying Qwen3.8-27B on an RTX 3090.
llama.cpp Setup:
- Used Q4KXL quantization with MTP/ngram speculative decoding.
- Enabled CUDA Graphs and Flash Attention.
- Achieved 45-50 tps with 150K context.
vLLM Comparison:
- Switched to vLLM via Docker, achieving 55-65 tps and 175K+ context.
- Drawback: inability to offload visual models (mmproj) to CPU (limited context with vision).
Quantization Discussion:
The user inquired about the quality difference of W4A16-AutoRound in vLLM versus Q4KXL for C/C++ and Python coding tasks, suspecting it falls between Q4KM and Q4KL.
More from coding & agent
- Open-Source Pinloop Uses Claude Code to Auto-Screen Millions of Job Posts — thisdudelikesAI · 2026-10-15
- Kimi Releases Workflow Tutorial #3: Swarm, Plugins and Local Project Sync — Kimi_Moonshot · 2026-10-15
- Dev bills $20,000 for an app built with $200 of Claude Code — IgorCarron · 2026-10-15
- Dan Shipper: run product and labs in parallel, even with a labs team of one — danshipper · 2026-10-15
- Gemini 4 Argon wrote 5,000+ lines of pure Python to build its own ray-tracing 3D engine — aitrendz_xyz · 2026-10-15
- Minecraft fully ported from Java to Rust, file by file, server and client included — Vjeux · 2026-10-15