Pocket AI Lab: open-source iOS app runs LLMs fully on-device with three backends
Ammoryyy · reddit · 2026-09-10
A developer released Pocket AI Lab, a free, MIT-licensed iOS app that runs LLMs entirely on-device — no cloud, no account, nothing leaves the phone.
Key details:
- Three backends side by side: MLX, llama.cpp and Core ML, picked per model. Paste a Hugging Face link and the app detects format, lists quants and predicts fit before downloading; you can also import .gguf files or pick from a curated list that updates without an app release.
- iOS memory engineering: third-party apps face a hard jetsam limit of roughly half the RAM. llama.cpp mmaps GGUF weights as file-backed pages that jetsam doesn't count — 8GB usable on an 11GB phone — while MLX and Core ML copy weights into dirty buffers and get 4.5GB. Same phone, same model: one backend runs it, another refuses. The app computes per-device budgets and refuses oversized loads instead of dying mid-run.
- On-device benchmarking: load time, prompt processing, tokens/sec with standard deviation, peak memory, thermal state, plus machine-checkable quality tasks. A build test downloads and benchmarks every catalog model your device can run, producing one report.
First-release bugs (two backends missing from the shipped build, long-prompt crashes, wrong memory budgets) are fixed. Code on GitHub, app on the App Store.
More from Infra
- colibri: Pure-C Zero-Dependency Engine Streams MoE Experts From Disk to Run Frontier Models Locally — JustVugg · 2026-09-10
- HBM seen topping 40% revenue share next year as Samsung challenges SK Hynix's lead — zephyr_z9 · 2026-09-10
- A quiet CPU shortage is hitting cloud users as hyperscalers shrug, devs report — mgill25 · 2026-09-10
- "Going for infinite energy": the Kardashev II mindset meme in AI circles — MarvinTBaumann · 2026-09-10
- Redis LangCache claims 90% LLM cost cuts; KV, prefix, prompt and semantic caching explained — blaizedsouza · 2026-09-10
- Raspbian creator laments $300 Raspberry Pi as AI demand prices out young programmers — evilsocket · 2026-09-10