Running a 27B Model on an iPhone
Prompt Engineering · youtube · 2026-07-15
The video breaks down how Prism ML compressed a **27B parameter Qwen-series model** to run on an **iPhone 17 Pro** at roughly **11 tokens/s**: - Used the **1-bit Bonsai** method to compress the model from about **54GB** to **3.9GB**. - Explained quantization basics: differences from 16-bit to 8/4-bit, ternary, and 1-bit, and why naive post-training quantization fails due to error accumulation. - Compared common open-source approaches: **llama.cpp**'s mixed precision, **GPTQ**'s calibration-aware quantization, and **AWQ**'s activation-aware quantization. These usually work up to ~4-bit but collapse at 2/1-bit. - Introduced **BitNet**-inspired quantization-aware training, end-to-end 1-bit/ternary solutions (including the LM head), multimodal support with a 4-bit vision tower, a **262K** context window, and latency reduction via speculative decoding. - Also touched on benchmarks, "intelligence density" per GB, shortcomings in tool calling, and the author's own 1-bit test results on an **M2 Max**: about **29–30 tok/s**, though it loops on harder tasks.
Related event: Quantized Qwen 27B Model Runs on iPhone(2 posts)→
More from Infra
- Local AI may pay back in 6–7 years and cut long-term costs by 30–40% — DavidLinthicum · 2026-07-21
- TSMC reportedly plans up to 10% chipmaking price hikes in 2027 — kimmonismus · 2026-07-21
- More open models and llama.cpp updates are coming, says Merve Noyan — mervenoyann · 2026-07-21
- Why adding a second LLM provider breaks more than the API surface — Ok_Extension6373 · 2026-07-21
- UK AI datacentres face backlash over heat, noise and land use — nordicinst · 2026-07-21
- Fluidstack raises $830M at $7.5B valuation as Anthropic backs a $50B compute buildout — rohanpaul_ai · 2026-07-21