Running Qwen3.8-27B-NVFP4 locally with vLLM at 1M context: full config shared
No_Night679 · reddit · 2026-09-16
A self-described beginner shared a working native (non-container) vLLM setup serving unsloth/Qwen3.8-27B-NVFP4 with a 1M-token context window, posting the full systemd unit.
Key settings: 4-way tensor parallelism with gpu-memory-utilization 0.91, fp8 KV cache, MTP speculative decoding with 3 speculative tokens, and hf-overrides injecting YaRN RoPE parameters (ropetheta 1e7, factor 4.0, originalmaxpositionembeddings 262144) to stretch context to 1,000,000 tokens. Qwen3 reasoning and qwen3xml tool-call parsers plus auto tool choice are enabled.
The author reports it runs well so far but is unsure whether the config can be tuned further — a directly copyable starting point for anyone attempting ultra-long context on local hardware.
More from Infra
- To dethrone Postgres, new databases must prioritize ecosystem integrations from day one — mgill25 · 2026-11-02
- ESP32-powered 4-inch touchscreen builds a TUI dashboard for monitoring a DGX inference cluster — CATLLM · 2026-11-02
- OpenRouter crosses 10 quadrillion annualized token run rate, doubling every 10.5 weeks — deedydas · 2026-11-02
- Palmer Luckey asks: can you use datacenter RAM for things prohibited by the operator's terms of service? — avaitopiper · 2026-11-02
- Disable MTP for 20-30% faster local LLM inference on limited RAM — InternalMode8159 · 2026-11-02
- Kioxia up 216% in a year but down 68% from its June 2026 peak — zephyr_z9 · 2026-11-02