Artificial Analysis and Liquid AI Launch On-Device Small Model Benchmark

Artificial Analysis, in partnership with Liquid AI, released an intelligence and inference benchmark for small models on mobile devices, testing models at 4-bit precision and below on iPhone 17 Pro and Galaxy S26 Ultra, with intelligence evaluations capped at 16K context to simulate phone memory constraints. The key finding: intelligence gaps between models are far smaller than efficiency gaps, making memory and speed—not capability—the real bottleneck for on-device deployment.

Confirmed

• The intelligence evaluation comprises five benchmarks covering function calling, knowledge, and reasoning; the inference benchmark measures speed, latency, and memory on real phones.
• Under the 16K context cap, Nanbeige4.2-3B (Reasoning) and LFM2.5-2.6B (Reasoning) tied for first with an average score of 63; Qwen3.5 9B led on BFCL and GPQA Diamond.
• Peak memory at 4K context on iPhone 17 Pro ranged from 0.4GB (LFM2.5-230M) to 6.9GB (Ornith-1.0-9B and Qwen3.5 9B), roughly a 19x gap; nearly 7GB severely squeezes system resources on 12GB phones.
• End-to-end time to generate 256 tokens ranged from 0.9s (LFM2.5-230M) to 26.7s (Falcon-H1R-7B), about a 30x gap; the two intelligence leaders were also notably slower.
• Token consumption to reach 60 points varied widely: Qwen3.5 9B (Reasoning) consumed 74.5M output tokens with 29% of generations exceeding the 16K window, while Gemma 4 E4B achieved similar scores with fewer tokens.

Why it matters

• This benchmark systematically quantifies the intelligence-efficiency tradeoff for small models on real hardware: models with similar scores can differ by an order of magnitude in memory and latency, which is far more useful for on-device model selection on mass-market 12GB phones.
• It also exposes the difficulty of fitting long chain-of-thought models on devices: generations exceeding the context window (29% for Qwen3.5 9B) get truncated, hurting usability.

2026-08-25 ~ 2026-08-25 · 9 related posts

• [source] Mobile Benchmark: LFM2.5 Leads Efficiency on iPhone 17 Pro — ArtificialAnlys · 2026-08-25
• Nanbeige and LFM2.5 tie for top mobile model score of 63 — ArtificialAnlys · 2026-08-25
• Qwen3.5 consumes 14x more tokens than Gemma 4 for similar mobile scores — ArtificialAnlys · 2026-08-25
• [source] Mobile inference spans 30x: LFM fastest at 0.9s, Falcon slowest at 26.7s — ArtificialAnlys · 2026-08-25
• [source] Mobile memory usage spans 19x: 9B models consume nearly 7GB peak — ArtificialAnlys · 2026-08-25
• Artificial Analysis launches benchmarks for small models on mobile phones — ArtificialAnlys · 2026-08-25
• Artificial Analysis partners with Liquid for mobile benchmarks — JosephJacks_ · 2026-08-25
• LiquidAI Partners for Mobile Small Model Benchmarking — JosephJacks_ · 2026-08-25

1 near-duplicate retellings: JosephJacks_