vLLM and NVIDIA Achieve Over 25K TPS/GPU for Qwen3.5 on GB200 Systems
AccBalanced · x · 2026-08-08
The vLLM team announced that through deep optimization in collaboration with NVIDIA, they achieved over 25,000 total tokens/s per GPU running Qwen3.5 on the GB200 NVL72 system.
The blog post details the optimization challenges for Qwen3.5's hybrid attention architecture (combining full-attention layers with Gated Delta Network layers). Key technical breakthroughs include:
- Blackwell-Optimized GDN Prefill Kernel: Delivering a 1.02x to 5.78x performance improvement over previous FLA/Triton implementations across various model sizes and batch shapes.
- Disaggregated Serving State Transfer: Solving the challenge of correctly transferring heterogeneous attention and GDN states between prefill and decode workers.
More from Infra
- Amazon Proposes Per-Request Validated Evidence Caching for Production RAG — _reachsumit · 2026-10-16
- LinkedIn's TransX report: sequence transduction for recsys, +6.02% CTR in A/B tests — _reachsumit · 2026-10-16
- llama.cpp PR boosts CPU inference speed with merged AVX2 tiled accumulators — jacek2023 · 2026-10-16
- Zero's next-gen IVM engine runs in a single DO, live-querying a 100GiB R2 database — threepointone · 2026-10-16
- Meme: driving to the chip factory in 2028 captures AI's compute anxiety — beffjezos · 2026-10-16
- One volunteer maintains the time zone file behind 4 billion phones — danbri · 2026-10-16