Running 20B MoE on Mac M4 with Only 500MB RAM at 40 TPS via SSD Streaming
netikas · reddit · 2026-08-10
The author explores extreme local LLM inference optimization on MacBook Air M4, focusing on minimizing memory footprint.
- Dilemma: Traditional small models (e.g., Qwen3 0.6B/1.7B) lack general capabilities, while quantized large models (e.g., Bonsai-27B) are painfully slow on Mac.
- Maple-Preview: A recently released 20B MoE model designed specifically for Mac, trained from scratch in ternary precision. It takes 5.31 GB (7.5 GB with 131K context).
- Extreme Optimization: By integrating Maple-Preview into Mference (a fork of turbo-fieldfare) to keep weights on SSD and stream MoE experts from disk, the author achieved 40 TPS generation using only 500MB of RAM on an Air M4.
- Capabilities: The model has limited world knowledge (English only) but handles simple questions with web search. Tool-calling benchmarks (Tau-2) are currently mediocre, but future agentic updates are expected.
More from Infra
- Discovered Materials Raises $9M to Hunt for Novel Chip Cooling Materials — TechCrunch AI · 2026-08-10
- 1M Token Context on Single RTX 3090 Achieved via KVarN Quantization — Anbeeld · 2026-08-10
- Choosing MiniMax H3 Quantization for RTX 5090: int8 vs nvfp4 — Zerozone000 · 2026-08-10
- MiniMax H3 Video Generation Stalls for 1 Hour on RTX 5090 — Johnwick1536 · 2026-08-10
- Offline KD Boosts Throughput 41% on Single H200, Slashes LLM Distillation Memory — MultiverseComputingCAI · 2026-08-10
- 50% higher costs: Why Chinese AI giants struggle to ditch Nvidia — pstAsiatech · 2026-08-10