Nemotron 3.5 Bolt 74B quantized hits ~169 tok/s on 16GB VRAM via llama.cpp
yehyakar · reddit · 2026-10-17
A Reddit user benchmarked NVIDIA-Nemotron-3.5-Bolt-74B-A9B (24GB GGUF, official IQ2XSS quant) on 16GB VRAM with llama.cpp, using identical 20k-token prompts with greedy decoding.
Key results:
- Stock v0.7.0 gives 168.8 tok/s; adding the smarter expert-cache eviction PR #30637 reaches 171.5 tok/s; upstream master and other builds are all within noise
- Strata-style hybrid mode (CPU computes cache misses) drops to 89–98 tok/s — on PCIe Gen5 x16, uploading a missing expert is faster than computing it on CPU; the profiler's only transferable piece (expert-routing profile, via PR #30594) barely matters since GPU cache hits 82% after warmup
- Magnitude doesn't support Bolt or models larger than VRAM
Tuning findings: prompt chunk 2048 reads 1960 tok/s but generates 10 tok/s slower (4096: 2178 read, 142 gen); expert cache maxes out safely at 7.5GB (7.8GB OOMs mid-generation); draft defaults (3 tokens/step, no confidence threshold) are best — thresholds 0.3/0.6 fall to 164/156 tok/s.
More from Infra
- Anthropic reportedly offers Blackstone 10%+ returns to fund AI chip deal — Polymarket · 2026-10-17
- Nvidia/Lumentum CW DFB laser array shown alongside Claude's rendered version — jwt0625 · 2026-10-17
- Debate: is tokens/sec per watt the right efficiency metric for AI? — 0xF2 · 2026-10-17
- kohya's sd-scripts and Musubi Tuner Support NVIDIA RTX Spark for Local LoRA Training at Launch — bdsqlsz · 2026-10-17
- Fund Managers Name AI Hyperscaler Capex the Top Systemic Credit Risk — SuB8u · 2026-10-17
- Vercel's tsc-ts ports tsgo back to TypeScript, runs 2-4x faster than tsgo — cramforce · 2026-10-17