W4A16 Quantization Boosts 30B Model Throughput by 4.5x on RTX 3090
mitchins-au · reddit · 2026-08-14
A developer released a W4A16 quantized version of NVIDIA's Nemotron 3.5 Lightning 30B-A3B model optimized for VLLM, allowing it to fit entirely on a single 24GB RTX 3090.
Tests show that W4A16 served via VLLM achieves several times higher throughput (around 4.5x at batch size 16) compared to the IQ4XS GGUF format on llama.cpp. Benchmarks indicate virtually no degradation in core capabilities like instruction following. This setup is ideal for consumer hardware handling batch labeling or agentic tasks where peak coding ability isn't critical.
More from Infra
- Three CoreWeave co-founders have sold $3.3B in shares since IPO amid credit worries — SumitGup · 2026-10-16
- Cherry pi on a Raspberry Pi: the tiny-model rumor developers are watching — sull · 2026-10-16
- CoreWeave's debt costs falling: DDTL spreads tightened from S+9.62% to S+2.25% — BenBajarin · 2026-10-16
- SoftBank lines up US IPOs for SB Energy and AI robotics spinoff Roze, valuations top $50B — nordicinst · 2026-10-16
- TRL reproduces Modal's mixed-precision async RL: plain GRPO collapses, SC+MIS holds 0.72 — QGallouedec · 2026-10-16
- Hugging Face scales Indic pretraining with 2.8T tokens and an open data pipeline — Hugging Face Blog · 2026-10-16