Full guide: running vLLM on Windows with Docker, WSL2, and an RTX PRO 6000

Demonicated · reddit · 2026-08-19

Windows users wanting vLLM usually settle for simpler tools like LM Studio; the author worked out a complete flow on an RTX PRO 6000 Blackwell (96GB) and let AI help write the guide. It runs Qwen3.8-27B as an OpenAI-compatible vLLM server via Windows 11 + WSL 2 + Docker Desktop, supporting vision, reasoning, tool calling, prefix caching, and MTP speculative decoding — with concurrent agent interactions at little throughput cost.

Key gotchas:

Reading a Windows bind mount across the WSL 9P boundary makes safetensors loading extremely slow — copy the model once into a Docker named volume (cache too) for much faster startups
Pin the vLLM 0.26.0 image tag so Docker doesn't silently reuse a stale latest cache
Don't start until all 18 checkpoint shards are down; a recommended 131K-context baseline is given (--max-num-seqs 2, --gpu-memory-utilization 0.92, chunked prefill, Qwen3 reasoning parser)
Reuse PowerShell variables in one session; download via the hf CLI

Original post →

Full guide: running vLLM on Windows with Docker, WSL2, and an RTX PRO 6000

More from Infra