Benchmark: Qwen3.8-27B-FP8 on single NVIDIA GH200 handles 262K context
MaziyarPanahi · x · 2026-08-15
Benchmarked Alibaba's Qwen3.8-27B-FP8 on a single NVIDIA GH200 using vLLM. With 10 concurrent real-time requests (max 16K output each, 262K total context), the first streamed events arrived within 10ms and all requests finished normally. Previous load tests showed throughput scaling from 148 tok/s (1 stream) to 3,395 tok/s (64 concurrent), achieving a 22.93× aggregate throughput increase without degradation.
Related event: Single GH200 runs Qwen 27B FP8: 10 concurrent streams, 10ms first token(5 posts)→
More from Infra
- TokenSpeed MLA 2 hits Vera Rubin: up to 3.44× faster MLA decode for Kimi K3 — zhyncs42 · 2026-10-16
- US vs Chinese hyperscalers: Americans fund and schmooze, Chinese demand your BoM and cap you at 7% margin — jwt0625 · 2026-10-16
- How LiteLLM routes one OpenAI-shaped API across every model provider — Mahmoud_Zalt · 2026-10-16
- OpenAI-Broadcom custom AI chip Jalapeño named a TIME Best Invention of 2026 — itsclivetime · 2026-10-16
- Developer live-runs continual pretraining of a Luganda LLM on Colab, 41% done with loss dropping fast — weskambale · 2026-10-16
- Jensen Huang: Nvidia charges ~$thousands per GPU yearly for software, a major revenue line — rohanpaul_ai · 2026-10-16