Local Nemotron 3.5 on one A100 hits 169 tok/s, processes 100 clinical notes for 2.4 cents
MaziyarPanahi · x · 2026-10-17
A hands-on benchmark of Nemotron 3.5 Lightning (30B total/3B active, BF16) on a single A100 80GB with vLLM 0.27.1, reasoning off and temperature 0: 169 tok/s per stream with 114 ms first-token latency, and all 100 discharge notes sent at once completed in 34 s (2,103 tok/s batch throughput). At $2.50/GPU-hour the whole run costs about 2.4 cents — all fully local, nothing leaves the machine. The test pushes back on the '100 tok/s or you won't use it' bar for local inference.
More from Infra
- India has only ~30,000 GPUs nationwide, 10-20x short of frontier model needs — CatAstro_Piyush · 2026-10-17
- MiniCPM-V 4.7 runs locally at 100 tok/s using only ~3 GB of memory — NielsRogge · 2026-10-17
- Power-management chips: the invisible layer deciding AI data center energy efficiency — Olivier__OG · 2026-10-17
- MiniCPM-V 4.7 runs a 35B multimodal model in 3GB of memory, fully on-device — alifcoder · 2026-10-17
- Full FP8 n-gram table visibly improves local Qwen3.8-Flash-Next at zero speed cost — sillynoobhorse · 2026-10-17
- NVIDIA's open-source AIPerf benchmarking tool and Triton GPU tutorials for inference measurement — techNmak · 2026-10-17