Run Qwen3.8-27B NVFP4 on RTX 5090: vLLM, 256K Context, 147 tok/s
AIFlow_ML · x · 2026-08-16
MiaAI-Lab releases a script to serve RadixArk/Qwen3.8-27B-NVFP4 with vLLM 0.27.x on a single RTX 5090 (32GB), with native 256K context, TurboQuant 4-bit KV cache (5.5 GiB), and MTP-3 speculative decoding, exposed as OpenAI-compatible API. Measured 160 tok/s single-stream generation, full 262,144-token context resident, and a patch fixing stock 0.27.1 garbling (0/15 vs 13/15 failures).
More from coding & agent
- Agents turn conservative as context grows: edits shrink from 150 to 30 lines — dbreunig · 2026-10-16
- ai& becomes an official built-in Cline provider with 13 open-weight models — DavidBennett__ · 2026-10-16
- x402 as a free GTM channel: agents discover 100+ tools, no API keys needed — 0xJeff · 2026-10-16
- Sam Altman: stop getting better at prompts, build loops and graphs that prompt for you — goyalshaliniuk · 2026-10-16
- POV ski game on real Matterhorn elevation data, built with Claude Opus 5.5 — techartist_ · 2026-10-16
- Poisoned SERPs broke rank tracking — a Claude MCP + Notion workaround — VeryWellVersed · 2026-10-16