Streaming 120B MoE Inference on Android

dai_app · reddit · 2026-07-18

Using a OnePlus 15R Android phone with roughly 11GB of available memory, the author ran several large models—including GPT-OSS-120B, Qwen 30B, and Gemma 26B—entirely on CPU and flash storage, achieving speeds of 1–5 tok/s.

Core Approach

Leveraging MoE architecture: only a few experts are activated per token, eliminating the need to keep all 60GB of weights resident in RAM
Using ODIRECT to read expert weights on demand from flash storage, combined with a small cache and asynchronous reads
Results are identical token-by-token compared to full-RAM inference, with consistency tests included in CI

Engineering Details

The underlying implementation isn't a fork of llama.cpp; it integrates via public callbacks and the gguf API
Adding a new MoE model requires just a single line in the registry
qwen3moe, qwen2moe, gemma4, and gpt-oss are currently supported

Benchmarks

GPT-OSS-120B: 1.3 tok/s
Qwen3-30B: 5.2 tok/s
Gemma-4-26B: 4.1 tok/s

Challenges

The real difficulty wasn't streaming reads, but Android reclaiming resident weights under memory pressure, causing continuous re-paging during generation. The author notes this consumed the bulk of the development effort. The project is Apache-2.0 licensed and offers a pre-compiled APK.

Original post →

Streaming 120B MoE Inference on Android

More from Infra