Achieving 181 tok/s on Qwen3.8 with 2x DGX Sparks via NVMe offloading
StartupTim · reddit · 2026-08-29
A developer achieved 181 tok/s aggregate throughput on Qwen3.8-Flash-Next using a 2-node NVIDIA DGX Spark (GB10) cluster, with single-stream decode at 30–50 tok/s.
Key Optimizations:
- Model & Quantization: Used RadixArk NVFP4 quantization and extended context to 512K via YaRN.
- NVMe Offloading: Left the 47.7 GB n-gram table on NVMe using madvise(MADVRANDOM) to fix read amplification, reducing GPU memory usage from 65GB to 41GB.
- vLLM Tuning: Enabled prefix caching (99% hit rate), MTP speculative decoding (k=3), and manual KV cache pinning to prevent OOM.
- Memory Management: Capped concurrent long prefills to stabilize host memory usage on the unified memory architecture.
More from coding & agent
- How to spot an AI-built frontend: it exposes everything the system knows — aryanXmahajan · 2026-08-29
- Developer Recommends Integrating WebMCP for Apps — kieranklaassen · 2026-08-29
- Hiten Shah shares workflow on using local AI for QA and bug fixing — msg · 2026-08-29
- Podcast preview: hnshah's librarian bot that organizes his GitHub repo — msg · 2026-08-29
- Hiten Shah on Running a Research, Marketing and Librarian Bot Team on Grok — msg · 2026-08-29
- Solving Agent Memory Conflicts: A Timestamp-Based Fact Replacement System — PrajwalTomar_ · 2026-08-29