MoEShift runs a 125B MoE on a 24GB Mac: 7.5GB peak memory at ~10 tok/s
dave-tro · reddit · 2026-10-30
A developer released the alpha of MoEShift, an open-source runtime that runs Qwen3.8 Flash Next 125B (ISTA-DASLab GSQ-RCO IQ3XXS quant, 76GB on disk) on a 24GB M4 Pro Mac mini with Safari and other apps open — 7.5GB peak process memory at 10 tok/s.
Vs Strata 0.2.7 (6k prompt, 256 output tokens): total inference 63.3s vs 129.9s; prompt processing 160.8 vs 61.8 tok/s; generation 9.9 tok/s with 7.5GB vs 14GB peak. Short-prompt generation: 10.44 vs 8.4 tok/s at 6.4GB vs 14GB.
How: loads experts from SSD on demand, caches hot experts in RAM, overlaps reads with GPU compute; built on llama.cpp and Lookahead Expert Scheduling, preserving original quants and expert routing, no speculative decoding.
More from Infra
- effect-harness on Cloudflare: run agent loops, tools and credentials on separate VMs — samgoodwin89 · 2026-10-31
- Google's Project Suncatcher boots TPU in orbit and beams back first AI response — sundarpichai · 2026-10-31
- Together AI launches Post-Training platform for agentic RL with tool-use environments — togethercompute · 2026-10-31
- General Compute raises $100M for inference capacity as CoreWeave stock hits 12-month low — YvesMulkers · 2026-10-31
- Analyst warns of unstable AI compute feedback loop as Kioxia's NAND price warning triggers flash crash — bookwormengr · 2026-10-31
- SOMA launches enterprise prompt-compression layer claiming 20%+ inference savings — markjeffrey · 2026-10-31