Achieving 181 tok/s on Qwen3.8 with 2x DGX Sparks via NVMe offloading
StartupTim · reddit · 2026-08-29
A developer achieved 181 tok/s aggregate throughput on Qwen3.8-Flash-Next using a 2-node NVIDIA DGX Spark (GB10) cluster, with single-stream decode at 30–50 tok/s.
Key Optimizations:
- Model & Quantization: Used RadixArk NVFP4 quantization and extended context to 512K via YaRN.
- NVMe Offloading: Left the 47.7 GB n-gram table on NVMe using madvise(MADVRANDOM) to fix read amplification, reducing GPU memory usage from 65GB to 41GB.
- vLLM Tuning: Enabled prefix caching (99% hit rate), MTP speculative decoding (k=3), and manual KV cache pinning to prevent OOM.
- Memory Management: Capped concurrent long prefills to stabilize host memory usage on the unified memory architecture.
Related event: Qwen3.8 hits 181 tok/s aggregate on dual DGX Spark nodes(2 posts)→
More from coding & agent
- A versioned bootstrap loader pattern for keeping AI-generated code always shippable — NickPassig · 2026-10-15
- Amazon blocks AI shopping agents; Maritime offers $1/month Android MCP workaround — Scobleizer · 2026-10-15
- Dev vibe-codes open-source newsletter platform, 83% cheaper than Beehiiv from $19/mo — marclou · 2026-10-15
- ERP AI launches Proto: 'headless SaaS' business software built for AI agents — Aiden_Tech_Ai · 2026-10-15
- Engineering managers need deep reading, not just agent summaries: 5 recommended resources — rseroter · 2026-10-15
- Adding motion graphics with just a prompt: the Claude Motion x ElevenLabs MCP workflow — CurieuxExplorer · 2026-10-15