Dev runs 120B models on iOS by streaming MoE weights from flash in llama.cpp

Chance-Resource-4970 · reddit · 2026-11-16

Developer Russell's Inpherio Live app (pending App Store review) turns an iPhone/iPad into a cloud-free local AI server. The trick: a modified llama.cpp streams MoE weights straight from flash while keeping active experts in RAM, bypassing iOS's 6GB per-app memory cap.

Benchmarks (Nov 2026):

Features: full OpenAI and Ollama API compatibility (chat, responses, embeddings, audio), local Vision (Qwen2.5-VL, Gemma 3, SmolVLM2), Whisper STT, Kokoro TTS, auto model loading with up to 3 warm models, and zero telemetry. Streaming large MoE models from storage will be a one-off purchase (Live Plus); everything else is free.

Original post →

More from coding & agent

coding & agent channel →