Dev runs 120B models on iOS by streaming MoE weights from flash in llama.cpp
Chance-Resource-4970 · reddit · 2026-11-16
Developer Russell's Inpherio Live app (pending App Store review) turns an iPhone/iPad into a cloud-free local AI server. The trick: a modified llama.cpp streams MoE weights straight from flash while keeping active experts in RAM, bypassing iOS's 6GB per-app memory cap.
Benchmarks (Nov 2026):
- gpt-oss 120B (58.5GB) on 16GB M1 iPad Pro: 4.5 tok/s from storage
- Qwen3 30B (17.3GB) on 12GB iPhone 17 Pro Max: 10.9 tok/s
- Qwen3 30B on 16GB M1 iPad Pro: 16.5 tok/s
- Qwen2.5 1.5B on iPhone 17 Pro Max: 58.3 tok/s in memory
Features: full OpenAI and Ollama API compatibility (chat, responses, embeddings, audio), local Vision (Qwen2.5-VL, Gemma 3, SmolVLM2), Whisper STT, Kokoro TTS, auto model loading with up to 3 warm models, and zero telemetry. Streaming large MoE models from storage will be a one-off purchase (Live Plus); everything else is free.
More from coding & agent
- shadcn boosts @uisdk: one-shot UI rebuild that's instantly E2E-testable by AI — shadcn · 2026-11-16
- AI agent deletes 43,599 lines of OpenClaw test code in one PR — vincent_koc · 2026-11-16
- JPMorgan shows Jarvis, its internal agent assembly line, at Interrupt NYC — LangChain · 2026-11-16
- Dev clarifies: Apple doesn't ban feature flags, only fooling App Review — talkaboutdesign · 2026-11-16
- Antigravity Matches Claude Cowork in Tests, While Gemini Tasks' Purpose Remains Unclear — sebajun2 · 2026-11-16
- One 16GB GPU runs songs, voice, images, 720p video and local RAG — all offline and open source — llama-of-death · 2026-11-16