Qwen 3.8 Flash Next on 2x CMP 170HX hits 840 tok/s at 8 concurrency, full recipe released
dead-fish · reddit · 2026-10-19
A detailed optimization writeup of Qwen 3.8 Flash Next (W4A16 AutoRound) on two CMP 170HX cards over PCIe Gen2 x4, using patched vLLM, pipeline parallelism, and native MTP speculative decoding. Measured aggregate decode: 226 tok/s single, 840 tok/s at 8 concurrent code requests (116 tok/s per request, 231ms TTFT); prefill 6,300–7,100 tok/s on 4k–64k prompts and 5,900 at 128k; prose decode 158→142 tok/s from 4k to 128k context. HumanEval 95.7%, GSM8K 96.4%, full 262,144-token window. Benchmarked via sparkDash protocol with medians of three runs. Full recipe, pinned Docker image, and patches on GitHub — reportedly the fastest published 2x CMP 170HX setup.
More from Infra
- vLLM cuts container startup time 14-18% with one uv bytecode flag — vllm_project · 2026-10-19
- Why neoclouds get acquired by outsiders, never by each other — AccBalanced · 2026-10-19
- Jev Router cuts coding agent costs 67-85% by routing tasks to cheap open models — HowDevelop · 2026-10-19
- Baidu's Virtual Head Attention cuts KV cache 4-7x with zero custom kernels — burny_tech · 2026-10-19
- Server memory spot growth slows to +0.6% as storage stocks fall 5-7% in a day — tengyanAI · 2026-10-19
- RTX Spark vs M5 Max: local Qwen 3.8 27B benchmark shows NVIDIA wins on compute — draginol · 2026-10-19