Qwen3.8-Flash-Next on 2x DGX Spark NVFP4: 50 t/s decode, 2,900 t/s prefill
-dysangel- · reddit · 2026-08-30
Redditor -dysangel- shares a full config for running Qwen3.8-Flash-Next NVFP4 on 2x NVIDIA DGX Spark (GB10): 49.7 t/s decode on structured output / 34.8 t/s prose, 2,875 t/s prefill on 11k tokens under TP2; single node does 35 t/s.
Stack highlights:
- vLLM PR #53896 release/qwen38next branch (model absent from main) plus a 2-file sm121 patch: add arch 12.1 to CMake lists and a 90-line software E2M1 conversion (GB10 lacks the hardware cvt instruction)
- 48GB PLE n-gram table served from internal NVMe via mmap — MADVRANDOM makes a 30x difference on hash-scattered lookups; a node-local CPU worker does gathers over ZMQ + pinned buffers + CUDA-IPC
- eager + MTP k=3 in a native venv, not Docker
Dead ends documented:
- PIEWISE cudagraphs cost 28% decode under MTP+TP2 (36 vs 50 t/s); they help single-node no-MTP (+80%)
- Docker silently degrades NCCL to TCP sockets (40% prefill); needs --privileged or IB passthrough
- RoCE device names are unstable across reboots and differ between identical machines; enumerate with ibdev2netdev and verify NET/IB via NCCLDEBUG=INFO
- flashinfer must be 0.6.18 on GB10; 0.6.17 crashes the NVFP4 MoE fallback kernel
- vLLM's shuffleInputRowsKernel has an uninitialized-permutation OOB read masquerading as a CUTLASS GEMM failure; a two-line guard fixes it
Next steps: block-diffusion drafting (DFlash/DSpark) and disaggregated decode onto a big-bandwidth Mac. MTP acceptance is 3.8 tokens/step on structured output but noticeably lower on prose — benchmark both.
Related event: Qwen3.8 hits 181 tok/s aggregate on dual DGX Spark nodes(2 posts)→
More from Infra
- engy.ai processes 50B+ tokens in a day as open-model inference demand surges — const_reborn · 2026-10-16
- New inference engine 'e' hits 10,500 tok/s on a single DGX Station — brandon_galang · 2026-10-16
- SGLang ships Day-0 support for FLUX 3 Image Klein: 10 reference images, BF16/FP8/NVFP4 variants — ying11231 · 2026-10-16
- NVIDIA's Nemotron 3.5 Super VL hits CoreWeave: 12B-active multimodal MoE at $0.25/M input tokens — wandb · 2026-10-16
- Chip equipment beats bull case: 2026 WFE seen at $153B, $278B by 2030 — BenBajarin · 2026-10-16
- Steam now launches in 5.23s, beating Epic's 5.47s — and vLLM gets roasted for slow startup — AlpinDale · 2026-10-16