Baseten tunes Nemotron 3 Ultra NIM serving, 2.5x more concurrent users on 4x B200

baseten · x · 2026-09-15

Baseten engineers detail the work behind serving NVIDIA's Nemotron 3 Ultra NIM efficiently: tuning caching, memory, parallelism, and decoding. On four B200 GPUs, those optimizations supported up to 2.5x more concurrent users while maintaining 50 TPS per user. A full engineering deep dive accompanies the announcement.

Related event: NVIDIA and Baseten Optimize Nemotron 3 Ultra Inference, Boosting Concurrency 2.5x on Four B200 GPUs(2 posts)→

Original post →

More from Infra

• NVIDIA releases NVFP4 quantized DeepSeek-V4.1-Flash, 665B params on HF — TheZachMueller · 2026-10-31
• Clover-35B-A3B Quantized to 29.4 GB Runs at ~42 tok/s on a 64 GB Mac — viveky259 · 2026-10-31
• Liquid Agent brings fully local LFM3 agent to Linux with one-command install — JosephJacks_ · 2026-10-31
• A fully local office AI stack: MiniCPM, Qwen3, LFM2.5 and Whisper — HowDevelop · 2026-10-31
• a16z: GPU rental prices up 106% in 3 months while token prices fall 58% — a16z · 2026-10-31
• vLLM nightly adds PLE disk offload, with a telling [Qwen4Exp] tag on the PR — rmhubbert · 2026-10-31

Infra channel →