Nemotron Bolt hits 70 tok/s on a 3090, inspiring MTP tuning on Qwen
samajhdar-bano2 · reddit · 2026-11-13
A Reddit user argues Nemotron models are the best open models around. On a 3090, vanilla llama.cpp with an IQ2 quant of the newly released Bolt 24G gave 70 tok/s; on a dual-R9700 coding rig, a Q8 deployment with CPU offloading and MTP drafters still hit 50 tok/s.
Inspired by Nvidia's MTP approach, he applied Tencent's DSpine drafter (and its GGUF llama.cpp fork) to his previously sluggish Qwen3.8-27B, reaching 54 tok/s at Q8KXL with mtp 15 and a 512k context pool. His takeaway: Nemotron's real value is teaching what good engineering looks like — techniques you can transfer to other strong models.
More from Infra
- Cloudflare hosts Thinking Machines' inkling 2 small on Workers AI at day-0 speed — michellechen · 2026-11-14
- Liquid-Codec decoder ported to Apple Neural Engine, 6x faster at 316x real time — JosephJacks_ · 2026-11-14
- AMD MI355X with FlyDSL kernels beats Blackwell running MiniMax M3.1 in vLLM — AnushElangovan · 2026-11-14
- Tencent paper: pruning neurons by routed contribution keeps 91-98% MoE performance — rohanpaul_ai · 2026-11-14
- AMD's hrx-loom stack: agent-written kernels beat llama.cpp across 51 LLMs on Strix Halo — jfowers_amd · 2026-11-14
- Microsoft Cloud tops $63B as AI buildout keeps scaling, says Nadella — thedealdirector · 2026-11-14