Running Ultra-Large MoE Models on Android
dai_app · reddit · 2026-07-19
The author experimented with a project called BigMoeOnEdge, running an MoE model over 5 times the size of available RAM on an Android phone.
Core Concept
- Leveraging the fact that MoE only routes to a few experts per token, the entire model isn't kept resident in memory.
- Using a routing hook to observe selected experts per layer, then directly streaming the corresponding tensor slices from flash storage, rebinding pointers, and computing.
- Other parameters never enter RAM; they are loaded per token on demand.
Benchmark Results
- On a standard Android phone, Qwen3.6-30B-A3B Q4KM:
- Direct mmap yields about 0.1 tok/s due to page cache thrashing.
- The expert streaming approach achieves 3.8 tok/s, and over 5 tok/s via CLI.
- gpt-oss-120b can also be loaded and generate text on the same device at roughly 1–2 tok/s.
Additional Info
- The project is based on llama.cpp, implemented via public eval callbacks and the gguf API. Upgrading upstream only requires a version bump.
- The author verified correctness: streamed output must be byte-for-byte identical to the fully resident version.
- The code and methodology are open-source under Apache-2.0.
Related event: Android Demo Runs 120B-Class MoE(2 posts)→
More from Infra
- MiniMax H3 runs fully on a Galaxy S26: 5-second video with audio in ~7 minutes — BidFunny4862 · 2026-10-27
- Unconventional AI acquires inference chip firm NeuReality in 1,000x efficiency bid — NaveenGRao · 2026-10-27
- Musk: robot RAM demand will dwarf everything else, and 100% of superintelligence compute goes to deep space — elonmusk · 2026-10-27
- RTX 5090 hits $7K as PC gaming community goes into 'denial' over AI boom — PAstynome · 2026-10-27
- Wan2GP lands on Pinokio: 1-click AI video on 6GB GPUs — cocktailpeanut · 2026-10-27
- AWS on agent prompt caching: explicit + implicit patterns to slash token costs — AWS ML Blog · 2026-10-27