NVIDIA: full-stack NIM tuning delivers 2.5x more concurrent users on Nemotron 3 Ultra

NVIDIAAI · x · 2026-09-15

• NVIDIA engineers published a deep dive on serving Nemotron 3 Ultra NIM: on a 4xB200 system, the optimized stack delivers up to 2.5x higher throughput versus an unoptimized baseline, hitting 1,997 tokens/s at a 50 TPS-per-user interactivity target.
• Optimizations include autotuned kernels, tensor parallelism, prefix and state reuse, scheduler/memory tuning, and MTP speculative decoding.
• NIM packages model- and GPU-aware serving choices into a deployable microservice, and NVIDIA AIPerf lets developers replay real workloads to pick the Pareto point meeting their latency SLO.

Original post →

More from Infra

• "Pace the frontier"? Investor says panic selling of AI chip stocks precedes biggest ramp ever — firstadopter · 2026-09-15
• Nvidia CMP 170HX Modded From 8GB to 64GB With 1.49 TB/s Bandwidth — _Boffin_ · 2026-09-15
• How Much Does Local LLM Inference Really Cost? A Dev Added an Electricity Calculator — giveen · 2026-09-15
• Hugging Face Rounds Up Which Open LLMs Are Best for On-Device Inference — NielsRogge · 2026-09-15
• Single Pure-C99 Inference Engine Runs Both BitNet Ternary and GGUF, No Python or CUDA — shifu_legend · 2026-09-15
• Dev weighs ChatGPT subscription via OAuth vs API pricing for a production RAG app — builtforoutput · 2026-09-15

Infra channel →