Claude Opus 5.5 misses 60% of hard phone budgets without real measurements

Mobile AI Factory: Building Specialized Language Models for Your Mobile Device

Evan Li, Krishna Deshpande, Wen Cheng, Shyamnath Gollakota

cs.CL

2026-11-10

Mobile AI Factory builds a quantized SLM per task and phone. On unseen tests it beats two five-shot frontier models on 5 of 10 tasks by up to 0.34, at 1/19 the GPU time.

What problem this solves

A small model on the phone can keep a medication list on the device and still answer with no signal. llama.cpp and MNN already run quantized models. The missing piece is a model tuned for one task and small enough for that phone's latency, memory, and energy budget.

Hand-tuning does not scale. One large app ships to more than ten thousand phone and tablet models, and about two new open base models show up every week. A fast setup on one handset can be the slow setup on the next. With llama.cpp, GPU prefill (reading the prompt before any tokens come out) is 5.4× faster than the CPU on the OnePlus 15 and 7.4× slower on the OnePlus 8 Pro. Qwen3-4B at 8-bit decodes 12.6 to 13.8 tokens per second on the first phone and 0.7 to 4.8 on the second.

Mobile AI Factory, from the University of Washington, takes a task spec and a target phone and returns a fine-tuned, quantized model that this phone's measured profile says will fit.

Method

Each phone is profiled once: prefill speed, decode speed, peak memory, and energy, across base model, bit width, engine (llama.cpp or MNN), and backend (CPU or GPU). Later tasks reuse that profile. Configurations over the latency, memory, or energy budget, and anything that crashed on that phone, are dropped. Each survivor keeps the engine and backend that decode fastest.

The pool is 27 variants: nine bases from 135M to 4B (SmolLM2, Gemma 3, Qwen3, Qwen3.5) at 4, 8, and 16 bits. They sit in five disk-size tiers, cut at 300 MB, 750 MB, 1.5 GB, and 2.5 GB. A 2B model at 4 bits and a 0.6B model at 16 bits can share a tier, because the phone feels bytes and speed. Search starts in the smallest tier and steps up only when the score has not gained more than 0.02 over 15 evaluations, or after 30 evaluations in that tier without hitting the goal. Collected training data moves up with it. Escalation is a fixed rule, so the orchestrator cannot open on the biggest model.

Claude Sonnet 5 proposes one change per turn: LoRA rank (only a low-rank adapter is trained), learning rate, weight decay, and epochs; more real rows from the task's own sources; or 25 to 600 synthetic rows from a local teacher, Qwen3.6-35B-A3B, aimed at the current failure types. A drop in score is rolled back. Training always restarts from the base weights, so rollback restores the previous best. On nested intent parsing, 88% of 104 interventions made the model worse and were undone.

The accuracy goal is the teacher's five-shot score, floored at 0.80 and capped at 0.99. If the run hits it quickly, the orchestrator may raise it by at least 0.005. Scores are taken on the quantized file the target engine will load. Under the same "4-bit" label, MNN's default export drops a fine-tuned SmolLM2-360M from 0.767 to 0.659, so MNN is scored with HQQ in blocks of 32 weights. The held-out set is capped at 1,000 rows, hidden from training, and guarded by four exact-match checks against leakage.

Results

Ten tasks, four phones: OnePlus 15, Galaxy S23, OnePlus 8 Pro, and Redmi 9A. Under the default goal, on the held-out set, the factory beat the stronger of Claude Sonnet 5 and DeepSeek v4 Flash (the better of zero-shot and five-shot, reasoning off) on seven tasks, by 0.037 to 0.357.

TaskFactoryFive-shot baseline
BC5CDR span F10.830 (Qwen3.5-0.8B 4-bit, 517 MB)Claude 0.606 / DeepSeek 0.508
Calendar to JSON0.811 (same 517 MB model)Claude 0.634 / DeepSeek 0.669
CLINC150, goal pinned at 0.990.937 (Qwen3-4B 8-bit)Claude 0.933, inside a 0.015 noise band
SMS spam, goal pinned at 0.990.961 (Qwen3-1.7B 4-bit)Claude 0.961

The last two rows are not default-goal wins. They match Claude only after the goal is pinned at 0.99. The held-out set steers every search step, so those scores run high. On an unseen test set, five tasks still beat both five-shot frontier models, by 0.024 to 0.342, and three more tie. Biomedical extraction there is 0.828 for the 517 MB model against Claude at 0.652.

The first fine-tune accounts for a median 82% of the gain over the untrained base. On biomedical extraction it moves the score from 0 to 0.705; search adds 0.125. Against that tier's own first fine-tune, search adds a median 0.082 on the smallest tier and 0.023 on the largest. Hyperparameters carry 41% of the search gain, synthetic data 40%, and extra real data 19%.

On that extraction task, tiered search used 8.6 training GPU-hours and $2.62 of orchestrator calls and stopped at 0.830. Training all 27 variants took 165 GPU-hours and $31.81, with a best of 0.859. The score is 0.029 lower, at about one nineteenth the GPU time. In the full sweep that same 0.8B model tops out at 0.835. Tiered search stopped because the goal was already met.

Call summaries on a Galaxy S23, memory cap 1,024 MB: ignoring the phone, the best model is an 8-bit 4B at ROUGE-L 0.482 and 4,082 MB on disk. It does not fit. The profile leaves 11 variants. The factory returns Qwen3-0.6B at 8-bit on MNN's CPU, at 0.469, which is 0.013 lower. Claude Opus 5.5, with high reasoning effort, full specifications, and no measurements, misses 25% of easy requirements, 25% of moderate ones, and 60% of hard ones. The engine is right 92% of the time. The CPU or GPU backend is right 75%.

Why it matters

A one-second budget for entity extraction wants a different deployment on every phone. OnePlus 15: Qwen3.5-0.8B, 4-bit, MNN GPU, F1 0.835. Galaxy S23: SmolLM2-360M, 4-bit, MNN CPU, 0.744. OnePlus 8 Pro: SmolLM2-135M, 4-bit, llama.cpp CPU, 0.719. On the Redmi 9A the fastest option takes 10 seconds, so nothing qualifies. Of 18 deployments tried on more than one phone, 15 need different configurations.

On hard budgets, measurements drop a median 88% of a phone's configurations before training starts. In 19 of 20 hard cases the largest model that fits is 0.8B or smaller.

Most of the accuracy is ordinary fine-tuning. Search picks the tier, the bit width, and the engine. On strict extraction, 517 MB beats five-shot frontier models. Intent classification and spam go the other way: with the default goal the factory stops at a 258 MB model, scoring 0.884 and 0.913, under Claude's five-shot 0.933 and 0.961. Pin the goal at 0.99 and it matches. Orchestrator spend on the ten main runs is $57.37. Training is the bill: 663 GPU-hours, two L40S GPUs per run.

Limitations

A person still has to supply data sources, the metric, and a verifier for outputs. That verifier is the most hand-built piece. Synthetic data pays off only when bad formats get rejected.

The pool is a fixed set of decoders from 135M to 4B, and training is LoRA only. No preference optimization, no reinforcement learning, no distillation. The teacher, Qwen3.6-35B-A3B, writes a few hundred rows per round. On seven tasks its five-shot score is under 0.80, which should turn synthesis off. The main runs left it on to measure the effect, and those tasks account for 67% of the gain credited to synthesis. On nested intent the teacher scores 0.02, yet 90% of the rows it writes pass the checker, and synthesis carries 60% of that task's search gain. The gate asks whether the teacher can solve items. The work is writing them.

Each configuration is a single run, and gaps inside 0.015 count as ties. Profiles omit NPUs. Every measurement is a cool phone, a 256-token prompt, four threads, and an idle device. Heat, other apps, and longer prompts may change which setups still meet the budget, and that was not measured. The main 4-bit and 8-bit scores were taken with llama.cpp on the training GPU. Copied to a OnePlus 15, eight tasks stay within 0.0014 of the server on CPU. Spam detection moves by 0.016.

Terms

Source

What people are saying

Related papers

All paper explainers