Running 30B Models on Aging 32-48GB GPUs Sparks Discussion
A Reddit user ran a 3.5-bit quantized model on two aging Quadro M6000s (48GB total) at 34.5 tok/s using Strata, while authors explained that quantization plus speculative decoding enable 30B models on 32GB GPUs.
2026-10-20 ~ 2026-10-21 · 2 related posts
- Running Strata on Two Ancient Quadro M6000s: 34.5 tok/s Decode After Lifting the Arch Restriction — jmuff98 · 2026-10-20
- How a 30B model fits on a 32GB GPU: quantization plus speculative decoding — TimDarcet · 2026-10-21