Full guide: running vLLM on Windows with Docker, WSL2, and an RTX PRO 6000
Demonicated · reddit · 2026-08-19
Windows users wanting vLLM usually settle for simpler tools like LM Studio; the author worked out a complete flow on an RTX PRO 6000 Blackwell (96GB) and let AI help write the guide. It runs Qwen3.8-27B as an OpenAI-compatible vLLM server via Windows 11 + WSL 2 + Docker Desktop, supporting vision, reasoning, tool calling, prefix caching, and MTP speculative decoding — with concurrent agent interactions at little throughput cost.
Key gotchas:
- Reading a Windows bind mount across the WSL 9P boundary makes safetensors loading extremely slow — copy the model once into a Docker named volume (cache too) for much faster startups
- Pin the vLLM 0.26.0 image tag so Docker doesn't silently reuse a stale latest cache
- Don't start until all 18 checkpoint shards are down; a recommended 131K-context baseline is given (--max-num-seqs 2, --gpu-memory-utilization 0.92, chunked prefill, Qwen3 reasoning parser)
- Reuse PowerShell variables in one session; download via the hf CLI
More from Infra
- Prediction: Opus 5.5-level models on laptops by 2027, backed by Qwen 27B data — cephaloform · 2026-10-16
- Oracle down 12% in a week, nearly 50% off June high as OpenAI exposure spooks market — alvelda · 2026-10-16
- US data center moratoriums top 400 — an AI bull explains why they're actually good for AI — kevinsxu · 2026-10-16
- Microsoft Research unveils Magnetar, the fastest zkVM at 10-39x speedups and post-quantum security — jedisct1 · 2026-10-16
- Bonsai 2 2B hits 120 tok/s on iPhone 18 Pro, outpaces Qwen 3.5 2B at half the memory — cephaloform · 2026-10-16
- Samsung cuts Q4 phone output as Nittobo shifts E-glass to AI-chip T-glass — zephyr_z9 · 2026-10-16