NVIDIA Details Qwen3.8-2.4T Deployment on GB300, Achieving >4K Tokens/s per GPU
PyTorch · x · 2026-08-21
NVIDIA published a technical blog detailing the deployment of Alibaba's latest open-weight model, Qwen3.8-2.4T-A95B, on the NVIDIA GB300 NVL72 platform.
- Model Specs: The model features 2.4T total parameters with 95B active per token, utilizing a MoE architecture and hybrid attention mechanism, supporting up to 1M token context window.
- Performance: Without additional tuning, the model achieves over 4,000 tokens/s throughput per GPU and over 350 tokens/s per user in FP8 precision.
- Tooling: Developers can leverage PyTorch-native fine-tuning and NVIDIA NeMo AutoModel to perform SFT or memory-efficient LoRA fine-tuning directly on existing checkpoints without model conversion.
More from Infra
- Nvidia's developer base hits 11 million, up ~3.7x from 3 million in 2021 — Beth_Kindig · 2026-10-16
- Muse Spark Open runs locally on a Mac M3 Ultra via llama.cpp: frontier model on desktop — pcuenq · 2026-10-16
- Muse-Spark open-weights drop: 816B-A42B MoE model gets 359GB MLX 3-bit build — ns123abc · 2026-10-16
- Meta launches Muse Spark Open, says it runs on two M5 Ultra Mac Studios with Apple's help — The Verge AI · 2026-10-16
- vLLM ships Day-0 LoRA RL for Muse Spark Open, lifting AIME from 0.56 to 0.85 — vllm_project · 2026-10-16
- DSpark draft model for Muse Spark Open proposes 7 tokens per pass with 23-42% acceptance — vllm_project · 2026-10-16