Running Ultra-Large MoE Models on Android
dai_app · reddit · 2026-07-19
The author experimented with a project called BigMoeOnEdge, running an MoE model over 5 times the size of available RAM on an Android phone.
Core Concept
- Leveraging the fact that MoE only routes to a few experts per token, the entire model isn't kept resident in memory.
- Using a routing hook to observe selected experts per layer, then directly streaming the corresponding tensor slices from flash storage, rebinding pointers, and computing.
- Other parameters never enter RAM; they are loaded per token on demand.
Benchmark Results
- On a standard Android phone, Qwen3.6-30B-A3B Q4KM:
- Direct mmap yields about 0.1 tok/s due to page cache thrashing.
- The expert streaming approach achieves 3.8 tok/s, and over 5 tok/s via CLI.
- gpt-oss-120b can also be loaded and generate text on the same device at roughly 1–2 tok/s.
Additional Info
- The project is based on llama.cpp, implemented via public eval callbacks and the gguf API. Upgrading upstream only requires a version bump.
- The author verified correctness: streamed output must be byte-for-byte identical to the fully resident version.
- The code and methodology are open-source under Apache-2.0.
Related event: Android Demo Runs 120B-Class MoE(2 posts)→
More from Infra
- Engram's random reads don't suit SSDs; CPU-memory over NVLink could serve all 72 GPUs — bookwormengr · 2026-09-11
- 80% of the DIY LLM inference hype posters have already quit — it's brutally hard systems work — abhijithneil · 2026-09-11
- Hugging Face's Ultra Scale Playbook: a free book on training LLMs on GPU clusters — mdancho84 · 2026-09-11
- Is inference latency becoming the biggest bottleneck for production AI agents? — Euphoric_Sea632 · 2026-09-11
- LLM Serving Metrics Thread: Why TPOT and Uptime Make or Break User Experience — abhijithneil · 2026-09-11
- PlanetScale launches sharded Postgres: 768 servers acting as one, 1PB scale — dhruv2038 · 2026-09-11