Nemotron 3.5 Bolt Hands-On: Tiny KV Cache Makes It Ideal for Small Multiuser Local Setups

1ncehost · reddit · 2026-10-19

A local-LLM enthusiast tested NVIDIA Nemotron 3.5 Bolt on a 128GB (4x MI100) rig. Key findings:

• Architecture: decoders are 2-bit quantized with NVFP4 for sensitive parts; 1 full-attention (GQA) layer per 5 linear-attention (Mamba2) layers — an even more aggressive ratio than Qwen3
• Extremely efficient KV cache: 30k context uses only 200MB (vs 500MB for Flash Next, 952MB for Qwen3 27B); 256k context is 1GB vs 3.4GB/8.5GB. The author calls the new Nemotrons the most KV-efficient new models except DS v4.1 Flash; the new Super shares the same attention
• Intelligence is roughly Qwen3.6 27B level, but the standout is a low hallucination rate — it tends to run diagnostics and thorough checks before acting, at about 2x the speed of 27B
• Best fit: small multiuser scenarios like small businesses, where millions of context tokens fit in memory; a single 5090 (32GB NVFP4) or a 128GB SFF box is ideal hardware

The author recommends it and hopes it signals the return of capable 70B-class models.

Original post →

More from Infra

• OCP attendees split on CPO/NPO light sources: external CW lasers (ELSFP) vs micro-VCSEL — bookwormengr · 2026-10-19
• Robot brain inference cost fell 84% in two months, from $2.36 to $0.38 per task — victor_explore · 2026-10-19
• SonicBoom v0.0.3: compiler-first AI runtime beats llama.cpp decode by 1.18x, calls for GPU benchmarking — nalaginrut · 2026-10-19
• M5 Ultra Mac Studio rumored with 512GB RAM; 6 units chained could run 2T-parameter models locally for ~$100k — SumitGup · 2026-10-19
• Apple Silicon's memory economics: M5 Ultra offers 5.6x better $/GB/s than DGX Spark for LLM inference — SumitGup · 2026-10-19
• Mac Studio M5 Ultra vs Dual DGX Spark: Long-Input Inference Benchmarks Show Split Verdict — GabGarrett · 2026-10-19

Infra channel →