Benchmarking Qwen3.8-27B on IGX Thor with RTX PRO 6000 Blackwell
ahstanin · reddit · 2026-08-25
The author shared detailed benchmark results for running Qwen3.8-27B-FP8 on an NVIDIA IGX Thor dev kit equipped with an RTX PRO 6000 Blackwell Max-Q (96GB).
Key Findings:
- Speculative Decoding Wins: Using the DFlash2 draft model improved throughput by 2.8x at batch 1 (126.3 vs 44.7 tok/s) and reduced TTFT by 3.3x. DFlash2 outperformed EAGLE.
- SSM State Dtype Pitfall: As a hybrid Gated DeltaNet model, Qwen3.8 has an extra GDN state pool. Running this pool in fp32 consumes too much memory, silently capping concurrency at 21 requests. Switching to bf16 restores full 32-request concurrency and doubles the KV pool with negligible accuracy loss on GSM8K (93.5% vs 94.0%).
- Reasoning Latency: Enabling 'thinking' mode adds 137ms to the first token latency; it's recommended to turn it off for voice applications.
- iGPU Beats RTX for Small Models: For STT and TTS workloads, the Thor iGPU outperformed the RTX 6000 in latency despite the latter's higher bandwidth.
More from Infra
- Arav Srinivas: On-device models critical for sensitive docs with SOTA OCR — AravSrinivas · 2026-08-25
- Arav Srinivas: Agentic inference must move to local hardware — AravSrinivas · 2026-08-25
- Data center boom drives a wave of gas power plant projects across the US — pstAsiatech · 2026-08-25
- 为何太空计算是突破 AI 能源瓶颈的终极解？ — McDonaghMatthew · 2026-08-25
- NVIDIA Super Acceleration for MiniMax H3 Video Generation — FaatmanSlim · 2026-08-25
- Nvidia loses perf/$ and perf/W crown as memory vendors consume CAPEX — zephyr_z9 · 2026-08-25