Running a 180B MoE Model on a 16GB MacBook: Experts Offloaded to SSD, 14× Faster Than llama.cpp

XiongChenyan · x · 2026-11-18

A new demo decouples MoE experts to SSD so a 180B-parameter model runs on a 16GB MacBook at decent decoding speed. The trick: retrain the router to predict experts, achieving 9GB peak memory and 14× faster decoding than llama.cpp.

Original post →

More from Infra

Infra channel →