Qwen3.8-2.4T-A95B open weights land on AWS: single 8×B300 node with vLLM
AWS ML Blog · rss · 2026-09-10
Alibaba's Qwen team released Qwen3.8-2.4T-A95B on August 12, 2026 — the first open-weights Qwen-Max-class model. An AWS ML Blog post walks through deploying it on SageMaker HyperPod with vLLM.
Architecture highlights
- 2.4T total parameters with 95B activated per token; fine-grained MoE with 512 routed + 1 shared expert, 10 routed activated per token
- Hybrid attention across 92 layers: 69 Gated DeltaNet (linear attention with bounded recurrent state) + 23 full-attention layers, keeping compute and memory bounded as context scales
- Native 262K context (extensible to 1M), 128K max output, native MTP draft heads for speculative decoding without a separate draft model
- Built-in reasoningeffort control (low/medium/high) to trade compute for reasoning depth
Deployment
- Runs on a single ml.p6-b300.48xlarge node: 8× NVIDIA B300 (2.1TB HBM3e total, 120 PFLOPS FP4)
- BF16 weights need 4.8TB; NVFP4 (W4A4) quantization compresses them to 1.2TB, fitting one node with KV-cache headroom
- The guide covers cluster provisioning through an OpenAI-compatible endpoint, including vLLM config for quantization, reasoning controls, tool calling, and MTP speculative decoding; HyperPod handles weight download, health checks, KEDA autoscaling, and node self-healing
Vendor benchmarks show strengths in PaperBench (93.0), IFBench (82.8), and terminal coding (86.6), comparable to frontier models, with headroom on repository-level tasks (SWE-bench Pro) — a credible self-hosted option for coding agents and research pipelines.
More from Infra
- AI gateway vs MCP gateway: do production agent stacks actually need both layers? — Purple_Morning_8735 · 2026-09-10
- Googler builds playable pickleball app on self-managed Spanner Omni in edge setup — rseroter · 2026-09-10
- LiteLLM vs TrustGate vs Portkey: picking an LLM proxy for production policy enforcement — OkInitial5068 · 2026-09-10
- GPU rental platform lium hits $964k monthly billing, 63% of rentals initiated by agents — markjeffrey · 2026-09-10
- First US rare earth magnet plant in 40 years breaks ground, 7,000 tons/year — MatthewChang · 2026-09-10
- LithosAI Opens Public API, Claiming Fastest and Cheapest Inference for Kimi K3 — sh_reya · 2026-09-10