How a 30B model fits on a 32GB GPU: search-based quant plus spec decoding
TimDarcet · x · 2026-10-21
Addressing skepticism that 30B params can't fit a 32GB GPU, TimDarcet explains the team used greedy and evolutionary search to find a custom quantization scheme minimizing both memory footprint and perplexity error, yielding two Pareto-optimal tradeoffs, plus speculative decoding. The Pareto chart shows EvoPress and Greedy-KL beating GPTQ, AWQ and Uniform NVFP4.
More from Models
- Epoch tests six frontier models on its own real work: reliable on defined tasks, short of full automation — dl_weekly · 2026-10-21
- Musk: Grok Bot is the fastest-growing product he's ever seen, enterprise rollout in weeks — XFreeze · 2026-10-21
- NVIDIA ships FP4-Local checkpoints of Qwen3.8-Flash-Next and GLM-5.3-Flash for local AI — NVIDIAAI · 2026-10-21
- Grok Users Hit by Context Bloat: Quota 'Evaporates Exponentially' Past a Threshold — alexcovo_eth · 2026-10-21
- Opus 5.5 spontaneously discusses Claude 3 Opus and model welfare in candid reply — repligate · 2026-10-21
- Surge AI's sudo L7: 60 real-repo tasks testing if coding agents act like staff engineers — echen · 2026-10-21