AWS blueprint: deploy NVIDIA Nemotron 3.5 Super VL (120B MoE) on EKS with Dynamo and vLLM
AWS ML Blog · rss · 2026-10-30
AWS published a reference implementation for serving NVIDIA Nemotron 3.5 Super VL on Amazon EKS, EKS Auto Mode, and SageMaker HyperPod EKS.
- Model: open multimodal MoE distilled from Nemotron 3 Ultra — 120B total / 12B active per token, hybrid Mamba-2 + attention + expert layers, RADIO vision encoder, 1M-token context, text/image/document/video input, BF16 and NVFP4 checkpoints, with weights, datasets, and post-training recipes published. Scores: 63.44% Hermes Bench, 87.05% LangChain agent eval, 80.8% CharXiv.
- Architecture notes: Mamba-2 layers keep constant-size state so memory doesn't grow like KV cache; a recursively shared MTP head enables native speculative decoding.
- Deployment: NVIDIA Dynamo supports aggregated or prefill/decode-disaggregated serving (vision encoding can split out too); minimum GPU configs range from 1× B300 (NVFP4) to 16× H100 (BF16).
- Performance methodology: validate clusters with NCCL all-reduce, measure TTFT/ITL/throughput with aiperf, and use concurrency-sweep curves for capacity planning — right-and-down is a win; rising ITL means add decode, rising TTFT means add prefill. Sample topology serves c32 concurrency (320 tok/s) within TTFT<600ms / ITL<100ms SLOs.
The aws-do-eks project automates the full pipeline from cluster creation to benchmarking.
More from Infra
- YC-backed VirajAero launches turbine that powers datacenters with 40% more power from same fuel — Scobleizer · 2026-10-30
- Proof of Agent rolls out 15-minute checkpoints anchored to Solana and Base, capping validation costs — adamamcbride · 2026-10-30
- Pooling $8K for a shared local LLM rig takes 3.3 years to beat a $20/mo ChatGPT sub — Any-Blacksmith-2054 · 2026-10-30
- 10 coding agents tested on local Qwen 3.8 27B: Pi wins on efficiency at 9.2k tokens per task — latent_lurker8587 · 2026-10-30
- Banks Shelve $5-7B Junk-Bond Sale for Anthropic-Leased Texas Data Center on Grid Audit — rohanpaul_ai · 2026-10-30
- Weight compression on the GPU: researcher proves his own instinct wrong — realDanFu · 2026-10-30