Local nemotron 3.5 lightning on one GPU hits 169 tok/s, 100 discharge notes in 34s
MaziyarPanahi · x · 2026-10-17
A dev ran nemotron 3.5 lightning locally on a single GPU for discharge notes: 169 tok/s per stream, 114 ms first token, all data on-machine; 100 notes batch-processed in 34s at 2 cents of GPU time. Respondent notes 100 tok/s is the usability bar — local now clears it.
More from Infra
- Self-hosting the Magpie gateway on a VPS to build your own OpenRouter — vista8 · 2026-10-18
- PyTorchCon 2026 highlights: MoE comm hiding, CUDA graphs, 10x data loaders — PyTorch · 2026-10-18
- Dan Ives: memory chip demand outstrips supply 13 to 1 after Korea trip — AccBalanced · 2026-10-18
- Building a local LLM station: is a cheap V100 still worth it over a 9700? — Numerous-Ad6217 · 2026-10-17
- Ben Lorica: Local AI Is a Routing Problem, Not Just a Privacy Feature — bigdata · 2026-10-17
- Bittensor expands subnet capacity to 2,500 miners, opening autonomous AI agent participation — bittingthembits · 2026-10-17