Apple's iPrefBench: mobile agents execute well (91.6%) but personalize poorly (47.7%)
apple · hf · 2026-10-28
Apple and Ohio State release iPrefBench, an executable iOS simulation benchmark testing whether mobile agents can infer missing user preferences from activity history or ask for necessary clarification. Grounded in a persona with 3,051 past events, it spans 130 underspecified tasks across 26 apps. Evaluating 7 frontier agents reveals high action completion (up to 91.6%) but low full-task success (up to 47.7%) — personalization lags far behind execution.
More from coding & agent
- Superwhisper taps Grok Bot, showing how businesses can ship products as conversational templates — XFreeze · 2026-10-28
- Sentient OS (YC F26) launches: on-device AI that learns your life overnight with a local LLM — testingcatalog · 2026-10-28
- Supermemory reverse-engineers how Claude Code's memory actually works — julianweisser · 2026-10-28
- GitHub's Spec Kit pushes spec-driven development: 5 commands to plan before coding agents build — Arindam_1729 · 2026-10-28
- How LangChain built its design agent: taste comes from context, not the model — LangChain · 2026-10-28
- Dev predicts in 5 years we'll laugh at today's juggling of terminal and agent harness tabs — DavidWells · 2026-10-28