Single GH200 runs Qwen 27B FP8: 10 concurrent streams, 10ms first token

On August 15, developer MaziyarPanahi published real-world tests: Alibaba's Qwen3.8-27B-FP8, running via vLLM on a single Lambda-hosted NVIDIA GH200, handled 10 concurrent real streaming requests with 16K-token max output per request and 262K context, with the first streaming event arriving within 10ms; a separate run showed roughly 3,300 tokens/s aggregate throughput at 64 concurrent requests. The takeaway: a single GH200 can now support ultra-long context, multi-way concurrency, and 10ms-class first-token latency simultaneously, providing a publicly checkable performance sample for single-GPU deployment of 27B-class models.

Confirmed

Environment: a single NVIDIA GH200 hosted on Lambda, vLLM inference framework, FP8-quantized Alibaba model
Concurrency test: 10 simultaneous real streaming requests with visible reasoning and final answers; 16K tokens max output per request, 262K context, first streaming event within 10ms
Throughput test: about 3,300 tokens/s aggregate across 64 parallel requests, with the first streaming response event likewise landing within 10ms

Unconfirmed

The model name appears as both Qwen3.8-27B-FP8 and Qwen3-8B-27B-FP8 across the posts; the material gives no official name
A repost relayed by MaziyarPanahi claims the model outperforms Opus 4.6, with no benchmark or test details attached and no independent verification

Why it matters

The FP8-plus-vLLM combination packs 262K context, 10-way concurrency, and 10ms-class first tokens into a single GH200, showing this class of service no longer depends on multi-GPU clusters
64-request throughput of 3,300 tokens/s means one card can absorb meaningful real traffic, giving smaller teams a direct reference for costing long-context model serving on single-GPU rentals

2026-08-15 ~ 2026-08-15 · 5 related posts

Primary sources

Qwen3.8-27B-FP8 on GH200: 10 Concurrent Streams, First Event in 10ms, 262K Context — MaziyarPanahi · 2026-08-15 00:48 (UTC+08:00)
Benchmark: Qwen3.8-27B-FP8 on single NVIDIA GH200 handles 262K context — MaziyarPanahi · 2026-08-15 01:07 (UTC+08:00)
Qwen3 hits 3.3k tokens/s on single Nvidia GH200 with 64 concurrent requests — MaziyarPanahi · 2026-08-15 05:56 (UTC+08:00)

Qwen3.8-27B-FP8 on GH200: 10 concurrent streaming requests, first token in 10ms — MaziyarPanahi · 2026-08-15
Qwen3.8-27B-FP8 Runs on Single GPU, Beats Opus 4.6 in Test — MaziyarPanahi · 2026-08-15
[source] Benchmark: Qwen3.8-27B-FP8 on single NVIDIA GH200 handles 262K context — MaziyarPanahi · 2026-08-15
[source] Qwen3 hits 3.3k tokens/s on single Nvidia GH200 with 64 concurrent requests — MaziyarPanahi · 2026-08-15

1 near-duplicate retellings: MaziyarPanahi