Running 180B Models on 48GB VRAM via NVMe Offload and Expert Pruning
EyalToledano · x · 2026-08-27
Combining NVMe offload (PLE) and expert pruning (REAP) makes it possible to run 180B-class models on a single 48GB GPU. Technical details include:
- mmap Optimization: On Apple's unified memory architecture, the 51GB n-gram table stays on SSD via mmap with page-in-touched-rows, incurring almost no performance cost.
- Performance: Qwen3.8-Flash-Next (Q4KXL) achieves 22 tok/s decode and 500-660 tok/s prefill, with only a 13% drop in decode speed as context grows.
- Double Compression:
- PLE offload removes the 25GB table from VRAM to SSD.
- REAP-256 pruning shrinks the 63GB expert pool.
- Result: Model size can be reduced to 40GB, enabling 180B models on 48GB VRAM.
This approach is particularly suitable for the MLX framework and allows further optimization by moving processes to the CPU.
Related event: Pruned Qwen3 180B MoE Runs on Laptop with 65GB Footprint(2 posts)→
More from Infra
- Guide: Deploy Agent Systems to AWS ECS with Terraform and GitHub Actions — kmeanskaran · 2026-08-28
- AMD ROCm 10.0.0 Released: Expanded Support Matrix and Easier Installation — AnushElangovan · 2026-08-28
- llama.cpp merges DFlash2 support: local convolution plus candidate selector — DjCanalex · 2026-08-28
- Gemma 4 MLX Challenge launches with 8% speedup on Mac — gajesh · 2026-08-28
- llama.cpp merges DFlash2 support: local convolution + candidate selector — jacek2023 · 2026-08-28
- Google explores stateless MCP for scalable agent tools — rseroter · 2026-08-28