Baseten tunes Nemotron 3 Ultra NIM serving, 2.5x more concurrent users on 4x B200
baseten · x · 2026-09-15
Baseten engineers detail the work behind serving NVIDIA's Nemotron 3 Ultra NIM efficiently: tuning caching, memory, parallelism, and decoding. On four B200 GPUs, those optimizations supported up to 2.5x more concurrent users while maintaining 50 TPS per user. A full engineering deep dive accompanies the announcement.
More from Infra
- NVIDIA Posts $96.2B Quarter: An AI Finance Model Separates Real Earnings From Future Promises — nikola_mr64990 · 2026-09-15
- AI's $1.1T data-center bet: productivity must grow 2.7x by 2030 to break even — nordicinst · 2026-09-15
- Own your embeddings: Weaviate + Ollama local vector search config that never leaves your network — philipvollet · 2026-09-15
- Chatbot injects 252k-token prompt every turn instead of RAG, dev asks to be talked out of it — louay_Sallakho · 2026-09-15
- MinIO: open-source S3-compatible object store with zero egress fees — thisguyknowsai · 2026-09-15
- Go-LLM repo details running Flash-Next on dual RTX 3090 via vLLM, with honest benchmarks — Motor_Ad16 · 2026-09-15