Right-size GenAI endpoints: SageMaker concurrency sweeps with Nemotron-3 Nano 30B
AWS ML Blog · rss · 2026-09-22
- AWS shows how to right-size generative AI endpoints using concurrency sweeps built into Amazon SageMaker AI Inference Recommendations — systematically increasing concurrent traffic (64→256→1024) to find where throughput plateaus and latency breaks.
- Deployment: NVIDIA Nemotron-3 Nano 30B (MoE, 3B active params) on ml.g7e.2xlarge via the native vLLM container; SMVLLMENFORCEEAGER=true is required for its Mamba-Transformer hybrid architecture, GPU memory util 0.85, prefix caching enabled.
- Benchmark: CreateAIBenchmarkJob with a 1024-in/256-out token workload (RAG-like), streaming on for TTFT, 1024 requests per level run sequentially.
- Analysis: watch output throughput plateau, p99 latency crossing SLA, widening p50–p99 spread (queuing signal), and TTFT; the saturation "knee" (256 concurrent in the demo) defines your safe operating region and instance count.
More from Infra
- Gas-for-AI-datacenters forum co-located with CoRL shows energy demand spillover — carlosdponx · 2026-11-12
- a16z podcast: Mike Volpi on Cisco's 70+ acquisition playbook and the AI buildout — a16z · 2026-11-12
- AMD to pass Intel in server revenue by mid-2027 as share gains 6 points a year — BenBajarin · 2026-11-12
- Study: Reflective Data Center Roofs Actually Cool Daytime Ground by 1.8-2.2°C — SumitGup · 2026-11-12
- Tsinghua's BaGuaLu-2 tames MoE expert-load spikes at runtime, accepted to SC26 — jiqizhixin · 2026-11-12
- antirez backs low-bit DeepSeek 4.1 Flash over Qwen 27B dense on Apple M5 Macs — antirez · 2026-11-12