SPIEval: Evaluating Large Language Models as Mobile Assistants over Scattered Personal Information
Junjie Ye, Zhuohui Sheng, Shaofan Liu, Yulun Zhu, Wenjie Fu, Dingwei Zhu, Ming Zhang, Yujiong Shen, Weichao Wang, Xin Zhao, Shihan Dou, Tao Gui, Qi Zhang, Xuanjing Huang, Pluto Zhou
cs.CL, cs.AI
2026-08-11
SPIEval benchmarks LLMs as mobile assistants: 250 tasks across five cognitive capabilities (reasoning, disambiguation, integration, preference inference, multi-intent decomposition), over 4,335 records in 10 apps, with 21 tools for multi-turn interaction. The strongest of nine models, GPT-5.5 (xhigh), reaches only 57.3 percent; 79 percent of failures come from wrong information localization, as models commit to a plausible record and stop verifying.
Using an LLM as a mobile assistant, a central difficulty is leveraging personal information scattered across apps to complete a request. "Save the suppliers from last week's trip" may require stitching together SMS, notes, and voicemail, while "call my wife" means identifying who she is and inferring from call history that she prefers video calls. Before this work no dedicated benchmark measured the task, leaving the capability a blind spot. SPIEval fills it.
SPIEval is a human-curated benchmark of 250 tasks evenly covering five cognitive capabilities: reasoning (multi-hop retrieval where each step depends on the prior), disambiguation (picking the currently valid record among several for one entity), integration (aggregating independent sources), preference inference (inferring habits never stated), and multi-intent decomposition (splitting a complex instruction into independent subtasks). Tasks run on 10 simulated apps (accommodation, alarm, contacts, meeting, notes, schedule, SMS, transactions, transport, voicemail) with a mean of 8.1 fields each and required-versus-optional distinctions, so records are partly filled and complete information must be assembled across apps, over 4,335 records total. Models get 21 tools: 11 retrieval tools (per-app plus global, supporting substring, regex, and fuzzy matching, field targeting, and pagination) and 10 execution tools. Scoring avoids LLM-as-a-judge and compares the model's final execution-tool calls against human gold answers at the parameter level; an outcome counts only if every parameter matches, and construction guarantees 100 percent human accuracy. Each task allows up to 50 turns and each setting runs three times.
Nine models range from GPT-5.5 (xhigh) at 57.3 percent to Kimi K2.6 (none) at 16.4 percent, with Gemini 3.1 Pro (high) 53.1, Claude Opus 4.8 (max) 52.3, and DeepSeek-V4-Pro (max) 46.5 in between. The three find-existing-information capabilities (reasoning, disambiguation, integration) average about 46 percent, while preference inference and multi-intent decomposition sit at roughly half that, because they require inferring what is not written. Raising reasoning effort lifts the average 13.8 points, but with large variance: GPT-5.5 gains 28.8, GLM-5.2 only 6.0.
The decisive finding is that localization is the bottleneck. Removing retrieval entirely (relevant records placed in the system prompt) jumps average accuracy from 35.5 to 66.8 percent, while returning all matches at once instead of paginated nudges it only from 35.5 to 36.0. Models are stuck on formulating queries that hit the target record. Of failures, 79 percent are wrong parameter values (a plausible-but-incorrect record chosen), and only 6 percent are missing required parameters. Models do not give up on hard tasks; they stop at the first plausible record and skip further verification. Across all 126,279 retrieval calls, 98.5 percent are plain substring queries, regex and fuzzy matching together fall under 2 percent, and only 9.5 percent target a specific field.
This turns a previously unmeasured capability (using scattered personal information as a mobile assistant) into a reproducible, parameter-level benchmark, and pins the failure on a concrete, fixable weakness: models do not lack reasoning, they lack localization, tending to commit to the first plausible record rather than keep searching. For teams building phone assistants or retrieval-augmented agents the signal is clear: rather than pile on reasoning, teach models to judge whether evidence is sufficient, to use field-targeted and fuzzy matching the tools already expose, and to retrieve one more round when uncertain.
The benchmark is hand-built by six NLP researchers over three months at a fixed scale of 250 tasks; coverage is broad but the count is limited, and generalization to real user phone data is not validated. All data is fictional, which avoids real-person issues but may miss the noise and long tail of real records. A 50-turn cap reflects mobile latency constraints but can truncate strategies needing more turns. The five capabilities have 50 tasks each, but execution operations are unevenly distributed (integration and multi-intent decomposition have more). Scoring is exact parameter matching, so equally valid non-gold answers are marked wrong, though 7.0 percent of parameters admit multiple valid values to soften this.