Ternary 1.58-bit Nemotron 30B-A3B GGUF Runs 368K Context on 12GB GPU
pmttyji · reddit · 2026-10-26
Hugging Face user boro64 released GGUF quantizations of Ternary Nemotron 3.5 Lightning 30B-A3B, a ternary (1.58-bit) quantized 30B MoE model (3B active parameters) that runs with 368K context on a 12GB GPU.
Reported benchmarks: 172 tok/s decode and 7000+ tok/s prefill. A single file works with both stock llama.cpp (validated on b11555+ and b11796) and the PrismML fork / Bonsai (validated on b10793).
More from Infra
- Reddit user builds $3360 Threadripper box with 256GB DDR4 to run large LLMs locally — sitytitan · 2026-10-26
- Top 1% of AI spenders direct over a third of AI budgets to raw GPU compute, Ramp data shows — rohanpaul_ai · 2026-10-26
- Running Qwen3.8B across 3 GPUs and 130GB RAM only hits 18 T/s — MageLD · 2026-10-26
- Used RTX 3090s Now Top $1500 on eBay as Owner Hunts a Replacement — like_a_tensor · 2026-10-26
- DeepSeek V4.1-Flash hits 49 tok/s decode on dual Radeon R9700 in local test — Longjumping-Elk-7756 · 2026-10-26
- Anthropic built Scion to replace git as AI-driven merges jumped from 2,500 to 30,000/day — bibryam · 2026-10-26