Running Nemotron-3.5-Bolt-74B MoE on a single RTX 3060 with 12GB VRAM
cephaloform · x · 2026-10-16
Developer DogukanUrker runs NVIDIA's Nemotron-3.5-Bolt-74B-A9B on a single RTX 3060 (12GB VRAM + 16GB RAM) with the full 262K context, at 24 tok/s decode and 500 tok/s prefill. NVIDIA's smallest official recipe asks for 32GB, but this 74B MoE with native 2-bit weights and only 8 attention layers fits on a 12GB card by pushing experts to RAM; 256K context costs just 1.2GB of KV cache.
- Key config: llama-server with -ngl 99 --n-cpu-moe 66 to offload expert layers, q80 quantized K/V caches, and -fa on for flash attention.
- The reposter calls the speeds "shockingly usable"; benchmarks to follow.
More from Infra
- Google's Willow chip cuts quantum errors 40x, raising Q-Day concerns for Bitcoin — jamestagg · 2026-10-16
- Vibe Coding a Data Pipeline Is Easy; Running It at Billions of Rows Is Not — Zachly · 2026-10-16
- Large data centers employ only ~50-150 permanent on-site staff, Microsoft says — brezina · 2026-10-16
- Prediction: Opus 5.5-level models on laptops by 2027, backed by Qwen 27B data — cephaloform · 2026-10-16
- Oracle down 12% in a week, nearly 50% off June high as OpenAI exposure spooks market — alvelda · 2026-10-16
- US data center moratoriums top 400 — an AI bull explains why they're actually good for AI — kevinsxu · 2026-10-16