Pocket AI Lab: open-source iOS app runs LLMs fully on-device with three backends
Ammoryyy · reddit · 2026-09-10
A developer released Pocket AI Lab, a free, MIT-licensed iOS app that runs LLMs entirely on-device — no cloud, no account, nothing leaves the phone.
Key details:
- Three backends side by side: MLX, llama.cpp and Core ML, picked per model. Paste a Hugging Face link and the app detects format, lists quants and predicts fit before downloading; you can also import .gguf files or pick from a curated list that updates without an app release.
- iOS memory engineering: third-party apps face a hard jetsam limit of roughly half the RAM. llama.cpp mmaps GGUF weights as file-backed pages that jetsam doesn't count — 8GB usable on an 11GB phone — while MLX and Core ML copy weights into dirty buffers and get 4.5GB. Same phone, same model: one backend runs it, another refuses. The app computes per-device budgets and refuses oversized loads instead of dying mid-run.
- On-device benchmarking: load time, prompt processing, tokens/sec with standard deviation, peak memory, thermal state, plus machine-checkable quality tasks. A build test downloads and benchmarks every catalog model your device can run, producing one report.
First-release bugs (two backends missing from the shipped build, long-prompt crashes, wrong memory budgets) are fixed. Code on GitHub, app on the App Store.
More from Infra
- Anthropic S-1: training infra edge cut per-task cost 40% and doubled revenue growth — BenBajarin · 2026-10-26
- Rewriting scaling law study: optimal synthetic data share rises from 10% to 32% with compute — heghbalz · 2026-10-26
- US states halt new data centers as 72% of Americans oppose local builds — YvesMulkers · 2026-10-26
- $5M Seed Round Can't Afford One DeepSeek Training Run: GPU-less AI Startup's Three Paths — FinanceYF5 · 2026-10-26
- HBM demand shifts: low-stack inference plans shelved, Rubin to go 12-high only — BenBajarin · 2026-10-26
- CircuitHub unveils The Grid, a factory-scale robot making PCBs in the US twice as fast as Shenzhen — garrytan · 2026-10-26