Qwen3.8 27B INT4 With 144K Context Runs on a Single RTX 3090 via vLLM
Altruistic_Heat_9531 · reddit · 2026-09-14
A Redditor shares a full recipe for running Qwen3.8-27B INT4 (AutoRound) with FP8 KV cache and a 147K-token context on one RTX 3090 via vLLM 0.27.1, beating llama.cpp's 25-30 tok/s. Key lessons: use AOT instead of JIT when compilation OOMs, watch the 5-6GB compilation cache, and set --gpu-memory-utilization 0.9475 with --mamba-cache-mode align for GDN models. BenchLocal eval scores 71/75 (94.7%) with reasoningeffort=low. Full command, Docker config, and benchmark scripts are open-sourced.
Related event: RTX 3090 Runs Qwen 27B Locally at 100 Tokens/s(2 posts)→
More from coding & agent
- Anthropic quietly ships server-side rule banning game rewrites in Claude Code — petrusenko_max · 2026-10-31
- Bootstrapping competitive edge with frontier tokens — curious_vii · 2026-10-31
- Element Forge: open-source editable AI VFX for real footage in ComfyUI — zitroffell · 2026-10-30
- Gym-Anything Adds Audit Agents That Try to Break Simulations With Legal Robot Actions — wellecks · 2026-10-30
- durable-object-migrator: zero-downtime exports for Cloudflare Durable Objects — samgoodwin89 · 2026-10-30
- This 5-question interview prompt makes Grok design your first two custom bots in minutes — nateliason · 2026-10-30