Overview of On-Device LLM Benchmarks

East-Muffin-6472 · reddit · 2026-07-11

This in-depth post summarizes the paper MELTing point: Mobile Evaluation of Language Transformers, focusing on evaluating the performance and user experience of LLMs on mobile and edge devices.

Experimental Setup

Test devices: iPhone 6 SE, iPhone 14 Pro, S23, Pixel 6a, Jetson Nano, Jetson AGX Orin.
Test models: tinyLlama, Llama-2, Gemma, Zephyr-3B, Mistral-7B.
Backends: MLC-LLM and llama.cpp, utilizing chat apps like MLChat and LLMFarm to simulate real-world usage.
A Raspberry Pi 4 acts as the control hub for task distribution, monitoring, and device interaction. Thermal cameras and system APIs log temperature and performance on the mobile devices.

Methodology & Metrics

Instead of just testing raw models, the study replicates real user experiences: opening chat apps, typing, generating, and continuous chatting.
They define QES (Quality Exp score), focusing on three pillars:
Responsiveness: Whether the device remains responsive during chat.
Stability: System stability across continuous, multi-turn conversations.
Temperature: Device heat levels.
They also explored split inference by running prefill on NVIDIA devices and offloading subsequent decode steps to phones via Wi-Fi 6.

Key Findings

The GPU pathway significantly outperforms CPU; iPhone results are particularly strong, largely because LLMFarm better utilizes Metal GPU acceleration.
Android devices are slightly faster than iPhones at CPU decode.
Q3/Q4 quantization is generally faster and more power-efficient on mobile.
During prolonged stress tests, throughput fluctuated on some devices, attributed to factors like DVFS.
Devices heated up noticeably, with surface temperatures reaching up to 47.1°C.

Original post →

Overview of On-Device LLM Benchmarks

More from Infra