Running a 27B Model on an iPhone
Prompt Engineering · youtube · 2026-07-15
The video breaks down how Prism ML compressed a 27B parameter Qwen-series model to run on an iPhone 17 Pro at roughly 11 tokens/s:
- Used the 1-bit Bonsai method to compress the model from about 54GB to 3.9GB.
- Explained quantization basics: differences from 16-bit to 8/4-bit, ternary, and 1-bit, and why naive post-training quantization fails due to error accumulation.
- Compared common open-source approaches: llama.cpp's mixed precision, GPTQ's calibration-aware quantization, and AWQ's activation-aware quantization. These usually work up to 4-bit but collapse at 2/1-bit.
- Introduced BitNet-inspired quantization-aware training, end-to-end 1-bit/ternary solutions (including the LM head), multimodal support with a 4-bit vision tower, a 262K context window, and latency reduction via speculative decoding.
- Also touched on benchmarks, "intelligence density" per GB, shortcomings in tool calling, and the author's own 1-bit test results on an M2 Max: about 29–30 tok/s, though it loops on harder tasks.
Related event: Quantized Qwen 27B Model Runs on iPhone(2 posts)→
More from Infra
- AWS tutorial: fleet-wide edge CV model lifecycle with SageMaker SDK v3, ~$0.07 per run — AWS ML Blog · 2026-10-27
- Qwen3.8-27B hits 40 tok/s on Jetson Thor with INT4 + MTP, accuracy beats FP16 — No-Dragonfly6246 · 2026-10-27
- Celesto launches CI runners nearly 90% cheaper than GitHub Actions, with agent sandboxes — aniketmaurya · 2026-10-27
- Supabase founders on rebuilding the backend for a billion AI-built apps — Latent Space · 2026-10-27
- AW-0 runs at 230ms on a single RTX 5090, 2.4x faster than autoregressive code planning — trevordarrell · 2026-10-27
- Site tracks US city council topics: data centers rank only behind pickleball courts — zakkohane · 2026-10-27