Developer Prunes 180B-class Qwen MoE to Fit in 48GB of Mac Memory

Developer Eyal Toledano ran a series of experiments compressing Qwen3.8-Flash-Next (a 180B MoE) down to consumer-Mac scale via expert pruning plus MLX quantization: pruned to 288 experts with 4-bit quantization, the model needs only 39GB of memory (vs 97GB originally) while keeping 91.5% HumanEval.

Confirmed

• Structural pruning (REAP) based on expert activity in real coding sessions cut experts per layer from 512 to 288; combined with native MLX 4-bit quantization, resident memory is 39GB, HumanEval holds at 91.5%, and it is smaller than the original q4 (m1, m3, m4, m6).
• REAP-384 intermediate result (25% of experts removed): footprint drops from 97GB to 80GB, retaining roughly 89% of original performance (m5).
• 8-bit variant Qwen3.8-Flash-Next-REAP-288-MLX-8bit: HumanEval 90.9%, 70GB resident memory, 27% lower memory than the original 8-bit quantization (m2).
• Combining NVMe offload (PLE) with pruning enables running a 180B-class model on a single 48GB GPU; on Apple unified memory, mmap + page-level optimizations are used (m7).
• A custom pure-MLX inference engine reaches 65 tok/s decoding with zero precision loss (targeting 100+), and Expert paging halves memory requirements (m8).
• Author correction: the 28 tok/s figure applies to high-bandwidth machines like M4 Max; MacBook Air will decode much slower but the model still fits in 48GB (m9).

Why it matters

• It demonstrates that very large MoE models can run locally on consumer Apple unified-memory hardware through structured pruning and aggressive quantization, sharply lowering deployment barriers.
• Pruning trades minimal benchmark loss for nearly halved memory, offering a reproducible methodology for MoE compression.
• Systems techniques like the pure MLX engine, Expert paging, and NVMe offload show further headroom for speed and memory savings.

2026-08-27 ~ 2026-08-28 · 9 related posts

• Pruning Qwen3 MoE to 65GB Fits a 180B-Class Model on a 128GB Laptop — EyalToledano · 2026-08-27
• Running 180B Models on 48GB VRAM via NVMe Offload and Expert Pruning — EyalToledano · 2026-08-27
• Qwen3.8-Flash-Next Fits on a 48GB MacBook: Pruning + SSD N-gram Table, 39GB RAM — EyalToledano · 2026-08-27
• [source] Two tricks squeeze Qwen3.8-Flash-Next onto a 48GB MacBook Air — EyalToledano · 2026-08-28
• [source] 180B-Class Qwen3.8 Model Runs on Just 39GB Memory — EyalToledano · 2026-08-28
• [source] Pruning Qwen to 288 experts enables 180B model on Mac with 39GB RAM — EyalToledano · 2026-08-28
• Qwen3.8 Variants: 8-bit Model Cuts Memory by 27% — EyalToledano · 2026-08-28
• Pure MLX Engine Hits 65 tok/s, Cuts Model Size in Half — EyalToledano · 2026-08-28
• REAP Pruning + MLX Quantization Squeeze a 180B-Class Qwen Into 39GB — 0xIlyy · 2026-08-28