Running Qwen 3.8 MoE on a single DGX Spark: A practical recipe
QuixiAI · x · 2026-08-29
Shantanu Goel shares a practical recipe for running the 135GB Qwen3.8-Flash-Next MoE model on a single NVIDIA DGX Spark (128GB VRAM). The solution uses NVFP4, MTP speculative decoding, and CUDA graphs. To fit the model, torch.fromfile is used to mmap the PLE table to NVMe, keeping it out of the unified memory pool while maintaining CUDA graph compatibility. The author also patched SGLang to bypass kernel gates restricting features to SM100, enabling FA4 paths on GB10 (SM121). Benchmarks were run in practical task configurations.
More from Infra
- Benchmark: Prefill/Decode split fails on commodity hardware — vllm_project · 2026-08-29
- Upgrading inference engine boosts decode by 43% on GH200 — colinmcnamara · 2026-08-29
- Debate on CUDA vs JIT DSL trade-offs and pitfalls — YouJiacheng · 2026-08-29
- Microsoft Fabric Variable Libraries: Eliminating hardcoding and configuration headaches — adnan_hashmi · 2026-08-29
- Ollama launches Claude integration to run local models in Claude Desktop — dr_cintas · 2026-08-29
- I have 500GB of Optane RAM, what AI projects can I run? — nosimsol · 2026-08-29