Streaming 120B MoE Inference on Android
dai_app · reddit · 2026-07-18
Using a OnePlus 15R Android phone with roughly 11GB of available memory, the author ran several large models—including GPT-OSS-120B, Qwen 30B, and Gemma 26B—entirely on CPU and flash storage, achieving speeds of 1–5 tok/s.
Core Approach
- Leveraging MoE architecture: only a few experts are activated per token, eliminating the need to keep all 60GB of weights resident in RAM
- Using ODIRECT to read expert weights on demand from flash storage, combined with a small cache and asynchronous reads
- Results are identical token-by-token compared to full-RAM inference, with consistency tests included in CI
Engineering Details
- The underlying implementation isn't a fork of llama.cpp; it integrates via public callbacks and the gguf API
- Adding a new MoE model requires just a single line in the registry
- qwen3moe, qwen2moe, gemma4, and gpt-oss are currently supported
Benchmarks
- GPT-OSS-120B: 1.3 tok/s
- Qwen3-30B: 5.2 tok/s
- Gemma-4-26B: 4.1 tok/s
Challenges
The real difficulty wasn't streaming reads, but Android reclaiming resident weights under memory pressure, causing continuous re-paging during generation. The author notes this consumed the bulk of the development effort. The project is Apache-2.0 licensed and offers a pre-compiled APK.
Related event: Android Demo Runs 120B-Class MoE(2 posts)→
More from Infra
- China’s AI arms race is increasingly defined by chips, data centers, and open models — BenBajarin · 2026-07-22
- Agent search bottlenecks are now about variance, not raw latency — rohanpaul_ai · 2026-07-22
- Gavin Baker argues Nvidia may be one of open source AI’s biggest supporters — GavinSBaker · 2026-07-22
- AI Power Demand Exposes US Energy Gap, Urging Shift from Scarcity to Abundance — bradneuberg · 2026-07-22
- Gavin Baker says Nvidia’s $630B figure would be system revenue, not all Nvidia’s — GavinSBaker · 2026-07-22
- A Firecracker-based platform says it can host 6,000 AI agents on one 256 GB server — maritime_sh · 2026-07-22