Benchmark details revealed: 30B-A3B BF16 on one A100 with vLLM 0.27.1
MaziyarPanahi · x · 2026-10-17
Follow-up with methodology: Nemotron 3.5 Lightning ran as 30B-A3B in BF16 on one A100 80GB with vLLM 0.27.1, reasoning off, temperature 0, on synthetic notes. Tok/s is decode tokens over decode time, median of 5 streams; the 2 cent cost is 34 s of GPU time at the listed $2.50/GPU-hour, GPU only.
More from Infra
- Self-hosting the Magpie gateway on a VPS to build your own OpenRouter — vista8 · 2026-10-18
- PyTorchCon 2026 highlights: MoE comm hiding, CUDA graphs, 10x data loaders — PyTorch · 2026-10-18
- Dan Ives: memory chip demand outstrips supply 13 to 1 after Korea trip — AccBalanced · 2026-10-18
- Building a local LLM station: is a cheap V100 still worth it over a 9700? — Numerous-Ad6217 · 2026-10-17
- Blogger's real-world test: new DeepSeek V4.1 Flash hits 99.41% cache hit rate on 500M tokens — AlchainHust · 2026-10-17
- Ben Lorica: Local AI Is a Routing Problem, Not Just a Privacy Feature — bigdata · 2026-10-17