Running Ultra-Large MoE Models on Android
dai_app · reddit · 2026-07-19
The author experimented with a project called **BigMoeOnEdge**, running an MoE model over 5 times the size of available RAM on an **Android phone**. ### Core Concept - Leveraging the fact that MoE only routes to a few experts per token, the entire model isn't kept resident in memory. - Using a routing hook to observe selected experts per layer, then directly streaming the corresponding tensor slices from flash storage, rebinding pointers, and computing. - Other parameters never enter RAM; they are loaded per token on demand. ### Benchmark Results - On a standard Android phone, **Qwen3.6-30B-A3B Q4_K_M**: - Direct `mmap` yields about **0.1 tok/s** due to page cache thrashing. - The expert streaming approach achieves **3.8 tok/s**, and over **5 tok/s** via CLI. - **gpt-oss-120b** can also be loaded and generate text on the same device at roughly **1–2 tok/s**. ### Additional Info - The project is based on **llama.cpp**, implemented via public eval callbacks and the gguf API. Upgrading upstream only requires a version bump. - The author verified correctness: streamed output must be byte-for-byte identical to the fully resident version. - The code and methodology are open-source under **Apache-2.0**.
Related event: Android Demo Runs 120B-Class MoE(2 posts)→
More from Infra
- Local AI may pay back in 6–7 years and cut long-term costs by 30–40% — DavidLinthicum · 2026-07-21
- TSMC reportedly plans up to 10% chipmaking price hikes in 2027 — kimmonismus · 2026-07-21
- More open models and llama.cpp updates are coming, says Merve Noyan — mervenoyann · 2026-07-21
- Why adding a second LLM provider breaks more than the API surface — Ok_Extension6373 · 2026-07-21
- UK AI datacentres face backlash over heat, noise and land use — nordicinst · 2026-07-21
- Fluidstack raises $830M at $7.5B valuation as Anthropic backs a $50B compute buildout — rohanpaul_ai · 2026-07-21