Running a 27B Model on an iPhone
Prompt Engineering · youtube · 2026-07-15
The video breaks down how Prism ML compressed a 27B parameter Qwen-series model to run on an iPhone 17 Pro at roughly 11 tokens/s:
- Used the 1-bit Bonsai method to compress the model from about 54GB to 3.9GB.
- Explained quantization basics: differences from 16-bit to 8/4-bit, ternary, and 1-bit, and why naive post-training quantization fails due to error accumulation.
- Compared common open-source approaches: llama.cpp's mixed precision, GPTQ's calibration-aware quantization, and AWQ's activation-aware quantization. These usually work up to 4-bit but collapse at 2/1-bit.
- Introduced BitNet-inspired quantization-aware training, end-to-end 1-bit/ternary solutions (including the LM head), multimodal support with a 4-bit vision tower, a 262K context window, and latency reduction via speculative decoding.
- Also touched on benchmarks, "intelligence density" per GB, shortcomings in tool calling, and the author's own 1-bit test results on an M2 Max: about 29–30 tok/s, though it loops on harder tasks.
Related event: Quantized Qwen 27B Model Runs on iPhone(2 posts)→
More from Infra
- Engram's random reads don't suit SSDs; CPU-memory over NVLink could serve all 72 GPUs — bookwormengr · 2026-09-11
- 80% of the DIY LLM inference hype posters have already quit — it's brutally hard systems work — abhijithneil · 2026-09-11
- Hugging Face's Ultra Scale Playbook: a free book on training LLMs on GPU clusters — mdancho84 · 2026-09-11
- Is inference latency becoming the biggest bottleneck for production AI agents? — Euphoric_Sea632 · 2026-09-11
- LLM Serving Metrics Thread: Why TPOT and Uptime Make or Break User Experience — abhijithneil · 2026-09-11
- PlanetScale launches sharded Postgres: 768 servers acting as one, 1PB scale — dhruv2038 · 2026-09-11