Qwen3.8-2.4T-A95B open weights land on AWS: single 8×B300 node with vLLM

AWS ML Blog · rss · 2026-09-10

Alibaba's Qwen team released Qwen3.8-2.4T-A95B on August 12, 2026 — the first open-weights Qwen-Max-class model. An AWS ML Blog post walks through deploying it on SageMaker HyperPod with vLLM.

Architecture highlights

• 2.4T total parameters with 95B activated per token; fine-grained MoE with 512 routed + 1 shared expert, 10 routed activated per token
• Hybrid attention across 92 layers: 69 Gated DeltaNet (linear attention with bounded recurrent state) + 23 full-attention layers, keeping compute and memory bounded as context scales
• Native 262K context (extensible to 1M), 128K max output, native MTP draft heads for speculative decoding without a separate draft model
• Built-in reasoningeffort control (low/medium/high) to trade compute for reasoning depth

Deployment

• Runs on a single ml.p6-b300.48xlarge node: 8× NVIDIA B300 (2.1TB HBM3e total, 120 PFLOPS FP4)
• BF16 weights need 4.8TB; NVFP4 (W4A4) quantization compresses them to 1.2TB, fitting one node with KV-cache headroom
• The guide covers cluster provisioning through an OpenAI-compatible endpoint, including vLLM config for quantization, reasoning controls, tool calling, and MTP speculative decoding; HyperPod handles weight download, health checks, KEDA autoscaling, and node self-healing

Vendor benchmarks show strengths in PaperBench (93.0), IFBench (82.8), and terminal coding (86.6), comparable to frontier models, with headroom on repository-level tasks (SWE-bench Pro) — a credible self-hosted option for coding agents and research pipelines.

Original post →

More from Infra

• $5M Seed Round Can't Afford One DeepSeek Training Run: GPU-less AI Startup's Three Paths — FinanceYF5 · 2026-10-26
• HBM demand shifts: low-stack inference plans shelved, Rubin to go 12-high only — BenBajarin · 2026-10-26
• CircuitHub unveils The Grid, a factory-scale robot making PCBs in the US twice as fast as Shenzhen — garrytan · 2026-10-26
• Ubicloud publishes "Fixing Tool-Call Accuracy in vLLM" — petrusenko_max · 2026-10-26
• Meta approves nonvoting Class C shares as 2026 capex guidance hits $135B–$145B — thione · 2026-10-26
• Unsloth Ships MiniMax-M3.1-Flash Quants: 2-bit Retains 91.5% Accuracy, Runs in 150GB RAM — danielhanchen · 2026-10-26

Infra channel →