Running 180B Models on 48GB VRAM via NVMe Offload and Expert Pruning
EyalToledano · x · 2026-08-27
Combining NVMe offload (PLE) and expert pruning (REAP) makes it possible to run 180B-class models on a single 48GB GPU. Technical details include:
- mmap Optimization: On Apple's unified memory architecture, the 51GB n-gram table stays on SSD via mmap with page-in-touched-rows, incurring almost no performance cost.
- Performance: Qwen3.8-Flash-Next (Q4KXL) achieves 22 tok/s decode and 500-660 tok/s prefill, with only a 13% drop in decode speed as context grows.
- Double Compression:
- PLE offload removes the 25GB table from VRAM to SSD.
- REAP-256 pruning shrinks the 63GB expert pool.
- Result: Model size can be reduced to 40GB, enabling 180B models on 48GB VRAM.
This approach is particularly suitable for the MLX framework and allows further optimization by moving processes to the CPU.
Related event: Developer Prunes 180B-class Qwen MoE to Fit in 48GB of Mac Memory(9 posts)→
More from Infra
- SK Hynix ties new US plant to looser chip equipment exports to China, sources say — pstAsiatech · 2026-10-15
- Perplexity's new embedding models: index with a 9B model, search with 0.6B — antoine_chaffin · 2026-10-15
- Unsloth ships GGUF quants for Nemotron 3.5: Bolt-74B runs in 24GB RAM — danielhanchen · 2026-10-15
- vLLM-Omni ships Day-0 serving for FLUX 3 Image Klein with BF16, FP8, NVFP4 support — vllm_project · 2026-10-15
- NVIDIA's Nemotron 3.5 Super VL lands training and inference support on Applied Compute — ypatil125 · 2026-10-15
- CoreWeave bond yield tops 16% as loans outlast customer contracts by ~2 years — tengyanAI · 2026-10-15