Achieving 181 tok/s on Qwen3.8 with 2x DGX Sparks via NVMe offloading
StartupTim · reddit · 2026-08-29
A developer achieved 181 tok/s aggregate throughput on Qwen3.8-Flash-Next using a 2-node NVIDIA DGX Spark (GB10) cluster, with single-stream decode at 30–50 tok/s.
Key Optimizations:
- Model & Quantization: Used RadixArk NVFP4 quantization and extended context to 512K via YaRN.
- NVMe Offloading: Left the 47.7 GB n-gram table on NVMe using madvise(MADVRANDOM) to fix read amplification, reducing GPU memory usage from 65GB to 41GB.
- vLLM Tuning: Enabled prefix caching (99% hit rate), MTP speculative decoding (k=3), and manual KV cache pinning to prevent OOM.
- Memory Management: Capped concurrent long prefills to stabilize host memory usage on the unified memory architecture.
Related event: Qwen3.8 hits 181 tok/s aggregate on dual DGX Spark nodes(2 posts)→
More from coding & agent
- ServiceNow Launches AI Passport: Cryptographically Signed Credentials for Every AI Agent — shashib · 2026-10-15
- ChatGPT Sites + MCP Events + dots: A Workflow-Triggered Agent Pattern — gabrielchua · 2026-10-15
- Dev reflects on hand-refactoring with Opus 4.6 instead of trusting agents — madeofAjala · 2026-10-15
- Why games are a real, taste-aware coding benchmark, not just a gimmick — vitaliychiley · 2026-10-15
- OpenUI 1.0 brings ChatGPT-style Intelligent UI to local open models, MIT-licensed — dr_cintas · 2026-10-15
- Proof Launches x401, an Open Protocol Letting AI Agents Prove Who They Act For — csuwildcat · 2026-10-15