Nemotron-3.5-Lightning runs on 16GB VRAM via ShimQuant patch
Daxfortuna · reddit · 2026-08-30
The author developed ShimQuant to optimize Nemotron-3.5-Lightning quantization. Due to tensor width misalignment with 256, standard tools failed to compress it effectively. By shimming rows during quantization and slicing at inference, the author achieved 3.07 bpw (11.77 GiB), enabling it to run on 16GB cards with a HumanEval score of 91.5%, matching larger builds. This requires a patched llama.cpp.
More from Infra
- Same 1,000 Cloudflare Durable Objects: $4,050/month cut to $22 with one trick — threepointone · 2026-10-16
- NVIDIA ships FP4-quantized Qwen3.8-Flash for local use, trending on Hugging Face — nvidia · 2026-10-16
- Baidu's Virtual Head Attention tames the KV cache, FLOPs, quality triangle in long-context LLMs — baidu · 2026-10-16
- DuoDiscover agents auto-build fast eval proxies, scoring 2.68x TPU kernel speedup — berkeley_ai · 2026-10-16
- Google Raises 2026 Capital Spending to Up to $205 Billion — emmanuelvivier · 2026-10-16
- Google open-sources AX, a Kubernetes-style orchestrator for billions of AI agent tasks — thisdudelikesAI · 2026-10-16