Transformers PR streams MoE experts from SSD to run a 180B model on a 32GB Mac
jomic01 · reddit · 2026-11-20
Transformers PR 50717 adds an expertsoffload option that keeps MoE expert weights on SSD and loads only the experts the router selects.
- On an M5 MacBook it lands within 6% of Strata on decode and slightly ahead on prefill, with numbers in the chart.
- The PR is unmerged and the API is still under discussion.
- The author says it's still too slow for how he uses coding agents.
Source: Hugging Face blog moe-expert-streaming.
More from Infra
- 220B MoE with full 262K context runs on RTX 3090s: 251 tok/s on 4 cards — QuixiAI · 2026-11-20
- Why IREN is down so bad: $15B capex vs $9B committed financing leaves $6B gap — tengyanAI · 2026-11-20
- Anthropic Gets Claude Prefill Running on AMD MI355X in Three Days — AccBalanced · 2026-11-20
- Customer cuts GitHub Actions spend 90%: 20-min job runs in 7 min on Celesto — aniketmaurya · 2026-11-20
- OpenSSL discloses CVSS 9.8 buffer overflow in EVP_EncodeBlock() — JacobHenner · 2026-11-20
- Anthropic's $2T valuation questioned: $104B net loss on $33B revenue in 9 months — deliprao · 2026-11-20