Right-size GenAI endpoints: SageMaker concurrency sweeps with Nemotron-3 Nano 30B

AWS ML Blog · rss · 2026-09-22

• AWS shows how to right-size generative AI endpoints using concurrency sweeps built into Amazon SageMaker AI Inference Recommendations — systematically increasing concurrent traffic (64→256→1024) to find where throughput plateaus and latency breaks.
• Deployment: NVIDIA Nemotron-3 Nano 30B (MoE, 3B active params) on ml.g7e.2xlarge via the native vLLM container; SMVLLMENFORCEEAGER=true is required for its Mamba-Transformer hybrid architecture, GPU memory util 0.85, prefix caching enabled.
• Benchmark: CreateAIBenchmarkJob with a 1024-in/256-out token workload (RAG-like), streaming on for TTFT, 1024 requests per level run sequentially.
• Analysis: watch output throughput plateau, p99 latency crossing SLA, widening p50–p99 spread (queuing signal), and TTFT; the saturation "knee" (256 concurrent in the demo) defines your safe operating region and instance count.

Original post →

More from Infra

• Reka EdgeQ VLM Runs Natively on Snapdragon 8 Elite NPU With 0.73s First Token — RekaAILabs · 2026-09-23
• CXL Disaggregated Memory Nears Commercial Deployment, Says Ben Bajarin After FMS Meetings — BenBajarin · 2026-09-23
• Sandboxing ComfyUI for AI agents in a VM with a single shared GPU remains unsolved on Windows — johnshedletsky · 2026-09-23
• Unsloth GGUFs run Qwen-Image-2.1 locally on as little as 6GB VRAM — danielhanchen · 2026-09-23
• H Company trains computer-use agents on SkyPilot: thousands of sub-second sandboxes — skypilot_org · 2026-09-23
• MiniMax H3 video gen runs locally on M5 Ultra: 768p in ~2m22s with optimizations — bakawolf123 · 2026-09-22

Infra channel →