Running a 180B MoE Model on a 16GB MacBook: Experts Offloaded to SSD, 14× Faster Than llama.cpp
XiongChenyan · x · 2026-11-18
A new demo decouples MoE experts to SSD so a 180B-parameter model runs on a 16GB MacBook at decent decoding speed. The trick: retrain the router to predict experts, achieving 9GB peak memory and 14× faster decoding than llama.cpp.
More from Infra
- Startup says Jensen Huang replied in 7 minutes, gave them a war room at NVIDIA HQ—9.6x local AI gains — bgmshana · 2026-11-18
- AMD-backed TensorWave signs 6-year deal to supply OpenAI 457MW on Helios racks — Beth_Kindig · 2026-11-18
- Nvidia's Evaporated $1T Is Likely Heading to Anthropic's IPO, Not Bonds — xiaosun86 · 2026-11-18
- NVIDIA Is No Longer Just a Supplier but an Investor, Customer, and Backer of the AI Ecosystem — sidjustice_ · 2026-11-18
- Polymarket Puts 90% Odds on Nvidia Beating Earnings in First Report Under Interim CEO — Polymarket · 2026-11-18
- Power is the bottleneck: NYC event explores the grid gap becoming a market of its own — ditzikow · 2026-11-18