How a 30B model fits on a 32GB GPU: quantization plus speculative decoding
TimDarcet · x · 2026-10-21
The author addresses skepticism about running a 30B-parameter model on a 32GB GPU, explaining that it relies on quantization and speculative decoding work. The post also includes a radar chart with axes normalized to 0-100 comparing relevant metrics.
Related event: Running 30B Models on Aging 32-48GB GPUs Sparks Discussion(2 posts)→
More from Infra
- FastVideo open-sources MiniMax FastH3 Omni with training code, 12x faster on Blackwell — multimodalart · 2026-10-21
- Anthropic's draft IPO filing: $16.3B H1 revenue, $80B net loss, $750B compute plans — kimmonismus · 2026-10-21
- dphnAI's distributed inference network served 32B DeepSeek tokens in 24h, ~6% of Venice's daily volume — Ronangmi · 2026-10-21
- Jensen Huang: every company will have two factories, one real and one digital — round · 2026-10-21
- DeepSeek V4 called 'god-tier infra, 4th-tier ML': 7 tricks stacked, 20 patches to fix them — teortaxesTex · 2026-10-21
- HSBC says CoreWeave shares to drop 30%+ as ROIC of 8.9% lags borrowing costs — GaryMarcus · 2026-10-21