Running 17.6GB Inkling 2 Mini on a 12GB phone at 3 tok/s
dai_app · reddit · 2026-11-22
Hours before launch, a developer ran the 17.6GB Inkling-2-Mini (IQ4NL, GGUF) on a 12GB-RAM Android phone costing $400.
- Engine: pre-release BigMoeOnEdge v0.47.1, which runs on any hardware with less RAM than the model requires
- Config: CPU decode, NPU (Hexagon) or CPU prefill
- Measured 2-3 tok/s for 180-token responses, peaking around 4 tok/s
- Claims speeds comparable to Strata, Basalt, etc., spanning Android/iOS phones, Macs, consumer laptops/GPUs to datacenter GPUs
- Weights: huggingface.co/ggml-org/Inkling-2-Mini-GGUF
More from Infra
- Speed-to-power dominates: on-site solar-plus-storage cheapest for AI data centers in 85% of US locations — mattbeane · 2026-11-22
- LexiEngine, an AMD-focused Rust inference engine, adds Inkling support — W61k3r · 2026-11-22
- 9B model hits 59 tok/s on a 16GB GPU, but tool calls lag at 58% BFCL — AIFlow_ML · 2026-11-22
- Can 16-32GB Local Models Match Closed-Source SOTA by End of 2025? — AnimalPuzzleheaded71 · 2026-11-22
- NVIDIA Paper Caches LLM Reasoning as CPU Code, 16-291x Cheaper — rohanpaul_ai · 2026-11-22
- Async RL's max_staleness=3 Doesn't Really Cap Staleness: 10-Framework Analysis — 青稞AI · 2026-11-22