NVIDIA: full-stack NIM tuning delivers 2.5x more concurrent users on Nemotron 3 Ultra
NVIDIAAI · x · 2026-09-15
- NVIDIA engineers published a deep dive on serving Nemotron 3 Ultra NIM: on a 4xB200 system, the optimized stack delivers up to 2.5x higher throughput versus an unoptimized baseline, hitting 1,997 tokens/s at a 50 TPS-per-user interactivity target.
- Optimizations include autotuned kernels, tensor parallelism, prefix and state reuse, scheduler/memory tuning, and MTP speculative decoding.
- NIM packages model- and GPU-aware serving choices into a deployable microservice, and NVIDIA AIPerf lets developers replay real workloads to pick the Pareto point meeting their latency SLO.
More from Infra
- Your AI agent would cost 260% more without prompt caching, Arena data shows — arena · 2026-10-30
- Nebius runs auction-driven GPU pricing, lifting rates ~20% amid tight supply to 2028 — BenBajarin · 2026-10-30
- macOS update silently re-adds eMudhra root certs users had removed — steipete · 2026-10-30
- Bittensor's Terza Inference routes one API across miners' models on subnet 103 — const_reborn · 2026-10-30
- SGLang Summit 2026 sells out sponsor booths and Full Passes ahead of Nov 12–13 event — ying11231 · 2026-10-30
- OpenRouter rebuilds provider routing: 46% fewer capacity errors, 16.9% faster TTFT — steipete · 2026-10-30