BigMoeOnEdge streams 91B+ MoE models on 12GB phones and 16GB Macs, hitting 25-30 tok/s
dai_app · reddit · 2026-10-20
BigMoeOnEdge is an adaptive streaming layer on top of unmodified llama.cpp: a hardware-aware planner plus expert cache/streaming lets MoE GGUF models far larger than RAM run on phones, laptops and desktops, with an OpenAI-compatible local server. Real-world numbers: Qwen3.6-35B-A3B (21GB) hits 10 tok/s decode + 100-300+ tok/s prefill on a 12GB Android phone (NPU) and 25-30 tok/s on a 16GB Mac (planner takes decode from 7 tok/s and a 2000-token prompt from 21s CPU to 3.4s GPU); Qwen3-Coder-30B-A3B runs at 23 tok/s on Mac; Qwen3.8-Flash-Next 125B (4.8× RAM) at 6.5-9 tok/s; even DeepSeek V4 Flash 284B (7.6× RAM) runs at 1 tok/s on a phone. The author has used it with OpenCode on Android and Mac and is considering a llama.cpp PR.
More from Infra
- MiniMax H3 video generation runs on a single RTX 3060 12GB via GGUF quantization — Ok-Inevitable8391 · 2026-10-20
- ByteDance, Alibaba, Tencent place over $35B in Nvidia chip orders amid supply crunch — pstAsiatech · 2026-10-20
- DeepSeek-V4.1-Flash tops Agent Arena open models, ranks #11 overall at $0.20/task — vista8 · 2026-10-20
- vLLM at 94,235 GitHub stars: the high-throughput LLM inference engine keeps climbing — vllm-project · 2026-10-20
- From-scratch GPT-2 124M pretrained on 2.48B tokens with 4x A40 — full writeup — upulbandara · 2026-10-20
- New Whale inference engine claims to crush vLLM and SGLang — EAccelerate_42 · 2026-10-20