Benchmarking Qwen3.8-27B on IGX Thor with RTX PRO 6000 Blackwell
ahstanin · reddit · 2026-08-25
The author shared detailed benchmark results for running Qwen3.8-27B-FP8 on an NVIDIA IGX Thor dev kit equipped with an RTX PRO 6000 Blackwell Max-Q (96GB).
Key Findings:
- Speculative Decoding Wins: Using the DFlash2 draft model improved throughput by 2.8x at batch 1 (126.3 vs 44.7 tok/s) and reduced TTFT by 3.3x. DFlash2 outperformed EAGLE.
- SSM State Dtype Pitfall: As a hybrid Gated DeltaNet model, Qwen3.8 has an extra GDN state pool. Running this pool in fp32 consumes too much memory, silently capping concurrency at 21 requests. Switching to bf16 restores full 32-request concurrency and doubles the KV pool with negligible accuracy loss on GSM8K (93.5% vs 94.0%).
- Reasoning Latency: Enabling 'thinking' mode adds 137ms to the first token latency; it's recommended to turn it off for voice applications.
- iGPU Beats RTX for Small Models: For STT and TTS workloads, the Thor iGPU outperformed the RTX 6000 in latency despite the latter's higher bandwidth.
More from Infra
- Next-gen GraalVM benchmarks: 4x faster startup, 2x less memory, no build step needed — RealGeneKim · 2026-10-15
- How Microsoft optimized Windows for Nvidia's Arm-based RTX Spark AI laptops — tomwarren · 2026-10-15
- How Microsoft optimized Windows for Nvidia's Arm-based RTX Spark laptops — The Verge AI · 2026-10-15
- Global smartphone shipments plunge 22% in worst quarterly decline on record as AI demand fuels memory costs — Polymarket · 2026-10-15
- SK Hynix drops 11% again, now below 3x FY2027 earnings as AI memory selloff deepens — JOBhakdi · 2026-10-15
- ASML Sells Zero Lithography Machines into Europe as US Hosts 75%+ of AI Compute — NinaDSchick · 2026-10-15