RTX 3090 Qwen3.8-27B deployment: vLLM outperforms llama.cpp
Lower-Ad6101 · reddit · 2026-08-28
A Reddit user shared a detailed optimization guide for deploying Qwen3.8-27B on an RTX 3090.
llama.cpp Setup:
- Used Q4KXL quantization with MTP/ngram speculative decoding.
- Enabled CUDA Graphs and Flash Attention.
- Achieved 45-50 tps with 150K context.
vLLM Comparison:
- Switched to vLLM via Docker, achieving 55-65 tps and 175K+ context.
- Drawback: inability to offload visual models (mmproj) to CPU (limited context with vision).
Quantization Discussion:
The user inquired about the quality difference of W4A16-AutoRound in vLLM versus Q4KXL for C/C++ and Python coding tasks, suspecting it falls between Q4KM and Q4KL.
More from coding & agent
- GPT-5.6 Sol reverses engineers 32-bit iOS games in an afternoon — gpt2chatbot · 2026-08-28
- Using Grok to automate job search: internship applications and study plans — brandon_galang · 2026-08-28
- GitHub project: Agents generate 3D assets and build games via code — const_reborn · 2026-08-28
- Replit introduces Intelligent Model Routing for automatic model selection — amasad · 2026-08-28
- Technical Question: How to run GPT on long-horizon tasks with continuous status checks? — BLUECOW009 · 2026-08-28
- A layered mental model for AI agent security — joshua_saxe · 2026-08-28