MoEShift runs a 125B model on 24GB Macs: M5 Pro hits 374 tok/s prefill
dave-tro · reddit · 2026-11-09
Developer davide-tro released MoEShift, an open-source inference backend for memory-constrained Apple Silicon Macs that runs large MoE models via SSD offloading. Benchmarks of Qwen3.8-Flash-Next 125B (ISTA-DASLab GSQ-RCO IQ3XXS, 76GB on disk) on two 24GB Macs:
- M4 Pro vs M5 Pro medians: short-prompt decode 9.84 → 14.69 tok/s; long-prompt prefill (6,060 tokens) 199 → 374 tok/s; long-prompt decode 9.57 → 13.14 tok/s; inference time 83.9s → 55.1s
- Peak memory stayed at just 8.5GB on both machines
- Strong prefill makes the model genuinely usable in chat/agent scenarios
Install via brew install davide-troiani/moeshift/moeshift; a setup wizard handles model selection and provisioning. The author says results are promising for the upcoming Qwen4 MoE and invites other Apple Silicon configs to share numbers.
More from Infra
- Tesla cuts Optimus AI6 chip memory to 144GB, a 33% reduction amid DRAM shortage — JOBhakdi · 2026-11-09
- Clairvoyance's Hivemind pools GPUs across PCs and Macs to run distributed AI workloads — draginol · 2026-11-09
- InferenceBench trend: frontier models' inference speedups grew from 3x to 12x in a year — maksym_andr · 2026-11-09
- Google announces TPU 8tx for confidential AI and TPU 8i Max for memory-hungry agents — testingcatalog · 2026-11-09
- PlayCanvas open-sources Matrix-style digital rain shader for 3D Gaussian splats — willeastcott · 2026-11-09
- Google ships four TPU variants in one year, courts ~100 external customers vs Nvidia — The Decoder · 2026-11-09