Developer prunes and quantizes 180B Qwen3.8 to run on 48GB Mac
Developer Eyal Toledano completed an experiment fitting a 180B-class MoE model on a consumer Mac: based on Qwen3.8-Flash-Next, structural pruning plus MLX quantization produced a 4-bit version that runs in just 39GB of memory while keeping HumanEval at 91.5%. The experiment shows that expert redundancy in large models can be safely compressed using real-world usage data, charting a viable path to running extra-large models locally.
Confirmed
- Pruning method: analyzed expert activation in real coding sessions and reduced each layer from 512 experts to 288 (m4).
- Qwen3.8-Flash-Next-REAP-288-MLX-4bit: native MLX 4-bit quantization, needs only 39GB of memory, smaller than the original q4, HumanEval held at 91.5% (m1, m3, m4).
- Qwen3.8-Flash-Next-REAP-288-MLX-8bit: 8-bit quantization, HumanEval 90.9%, 70GB resident memory, reportedly a 27% reduction in memory footprint versus standard 8-bit quantization (m2).
- Performance: Q4 decodes at about 28 tok/s on high-bandwidth machines (e.g., M4 Max); Q8 needs roughly 50GB of memory (m3, m5).
- Author correction: initially stated it ran on a 48GB MacBook Air, later clarified that the 28 tok/s figure applies to high-bandwidth machines like the M4 Max; the MacBook Air has lower bandwidth and much slower decoding, though the model still fits within its memory budget (m5).
Why it matters
- This pipeline of "expert activation analysis + structural pruning + MLX quantization" demonstrates the real-world feasibility of running 180B-class models on consumer hardware with minimal benchmark loss.
- The 8-bit and 4-bit versions give users with different memory budgets two options: 39GB vs. 70GB resident memory.
2026-08-27 ~ 2026-08-28 · 5 related posts
Primary sources
- Pruning Qwen to 288 experts enables 180B model on Mac with 39GB RAM — EyalToledano ·
- Two tricks squeeze Qwen3.8-Flash-Next onto a 48GB MacBook Air — EyalToledano ·
- Qwen3.8-Flash-Next Fits on a 48GB MacBook: Pruning + SSD N-gram Table, 39GB RAM — EyalToledano · 2026-08-27
- [source] Two tricks squeeze Qwen3.8-Flash-Next onto a 48GB MacBook Air — EyalToledano · 2026-08-28
- 180B-Class Qwen3.8 Model Runs on Just 39GB Memory — EyalToledano · 2026-08-28
- [source] Pruning Qwen to 288 experts enables 180B model on Mac with 39GB RAM — EyalToledano · 2026-08-28
- Qwen3.8 Variants: 8-bit Model Cuts Memory by 27% — EyalToledano · 2026-08-28