TokenSpeed serves Qwen3.8-Flash-Next at 685 tokens/s per user on 4x NVIDIA GB300 with NVFP4
Alibaba_Qwen · x · 2026-10-14
LightSeek Foundation's TokenSpeed hits 685 output tokens/s per user serving Qwen3.8-Flash-Next on 4x NVIDIA GB300 (NVFP4), with Alibaba's Qwen team amplifying the result.
The writeup details optimizations for the new architecture: efficient sparse selection and attention for Qwen Sparse Attention (QSA), fusing computation across residual boundaries, cutting repeated work in MTP speculative decoding, and programmatic dependent launch to overlap kernel preparation. The model is an early Qwen4 preview: multimodal MoE with 125B total / 6B active params, plus 51B n-gram embeddings and a 4B MTP module, using hybrid Gated DeltaNet attention and four-branch Gated Residual.
More from Infra
- Mesa merges YOLOX with PyTorch support in Torx NPU driver — Fcking_Chuck · 2026-10-14
- Hugging Face launches Kernel Hub to distribute compute kernels as first-class artifacts — ariG23498 · 2026-10-14
- Benchmarking disaggregated LLM serving: KV cache transfer and energy costs can erase the gains — burkov · 2026-10-14
- jitllm moves a running LLM's layers between GPUs mid-reply in 8ms with zero recompute — web3samy · 2026-10-14
- Micron hikes 2027 HBM prices as memory nears 23% of PC build cost — tengyanAI · 2026-10-14
- Beff claims TSUs paired with GPUs will deliver 100x perf-per-watt via disaggregated inference — beffjezos · 2026-10-14