Running Qwen3.8-Flash-Next on six 3090s: 7 users at 262k context with a patched vLLM
flynth92 · reddit · 2026-10-12
A r/LocalLLaMA user shares a working vLLM branch plus benchmarks for serving Qwen3.8-Flash-Next on six RTX 3090s in a consumer PC (no EPYC, no P2P, four cards over Thunderbolt).
Results (int8 KV + GPTQ-INT4): prefill at 6,574 t/s, 37.8s time-to-first-token on a 250k prompt (vs 118s on llama.cpp-multigpu), 57.6 t/s single-user decode, and 206 t/s aggregate across 7 concurrent users at 262k context — 7.33 resident sessions vs 5 previously.
Key fixes: upstream vLLM refuses to start this setup at all (no PP>1, TP restricted to 1/2/4/8, 24GB card can't hold TP=1). The branch also puts the 102GB n-gram (PLE) table on SSD to save 108GB of RAM — matching a PR that landed upstream today — and replaces fp8 KV, which crashes on Ampere, with int8.
What didn't work: NVFP4 is 10–12% slower than GPTQ on Ampere (Marlin weight-only, no FP4 acceleration); per-user decode under pipeline parallelism drops with user count (57→29), while llama.cpp grouped decode stays flat — so vLLM for prefill/single-user speed/seats, llama.cpp for equal per-user speed. An AMD path via PP=3 on Radeon V620 exists too. Code was LLM-written but validated with needle tests and kernel oracles, one commit per change.
More from Infra
- Day 280/365 of GPU Programming: Diving into NCCL and How PyTorch Calls AllReduce — NandoDF · 2026-10-12
- FT: China rapidly builds data centers in energy-rich hinterland to power AI race — EleanorOlcott · 2026-10-12
- Lumentum Optics Sold Out Through Early 2029, Yet Only 44 of 149 AI Stocks Rose Last Week — demian_ai · 2026-10-12
- HF Engineer Runs pi-web on DGX Spark to Steer Local AI Agents From His Phone — lhoestq · 2026-10-12
- Open-source leaderboard whatmodel.sh shows the best local models your machine can run — jakecoolguy · 2026-10-12
- Grid becomes AI's new bottleneck: Entergy warned Amazon of peak-time cutoffs, AWS built Project Falcon — kimmonismus · 2026-10-12