Qwen3.8 27B INT4 With 144K Context Runs on a Single RTX 3090 via vLLM
Altruistic_Heat_9531 · reddit · 2026-09-14
A Redditor shares a full recipe for running Qwen3.8-27B INT4 (AutoRound) with FP8 KV cache and a 147K-token context on one RTX 3090 via vLLM 0.27.1, beating llama.cpp's 25-30 tok/s. Key lessons: use AOT instead of JIT when compilation OOMs, watch the 5-6GB compilation cache, and set --gpu-memory-utilization 0.9475 with --mamba-cache-mode align for GDN models. BenchLocal eval scores 71/75 (94.7%) with reasoningeffort=low. Full command, Docker config, and benchmark scripts are open-sourced.
More from coding & agent
- OpenViking launches as an open-source context database for AI agents — adnan_hashmi · 2026-09-14
- Git worktrees ~80% faster via APFS/btrfs/XFS/ReFS folder clones, with big disk savings — steipete · 2026-09-14
- Dev burns ~80% of a 20x AI sub debugging cascading AWS failures — generativist · 2026-09-14
- Microsoft's AgentRx paper pinpoints where long agent traces really fail — JohnAlexander · 2026-09-14
- Dev tip: route Claude Code subagents to Opus to save premium usage — lydiahallie · 2026-09-14
- Engineer maps NFL offense onto agent architecture: QB as main agent, WR as tools — philipkiely · 2026-09-14