AndroidWorld: Why mobile agent benchmarks are broken and how to fix them
East-Muffin-6472 · reddit · 2026-09-02
The AndroidWorld paper exposes a fundamental flaw in previous Android agent benchmarks: static test sets allow models to memorize answers rather than demonstrate capability.
Key Innovations:
- Parameterized Task Templates: Uses templates with variables (e.g., "Create a calendar event for {day} at {hour}") to generate millions of unique task variations per run, eliminating memorization.
- Automated Verification: Runs on a real Android emulator with built-in initialize(), issuccessful(), and teardown() functions. Success is determined by inspecting OS state via ADB, removing the need for human judges and ensuring reproducibility.
Results (M3A Agent + GPT-4 Turbo):
- AndroidWorld (116 tasks/20 real apps): 30.6% success (Human: 80%).
- MobileMiniWoB++ (Web tasks): 68% success (Human: 100%).
- Latency: Agents take 3.9 min/task on average, 3× slower than humans.
Key Finding: Agents appeared broken with fixed random seeds but solved the same tasks with variable seeds, proving static benchmarks often measure unlucky parameter combinations rather than true agent capability.
More from Embodied
- Tesla Cybercab Launching This Week, Challenging Waymo — JOBhakdi · 2026-09-02
- Upcoming meetup for women in robotics in India — alysha_lobo · 2026-09-02
- GoatRemote update: AI voice commands for Mac via Siri Remote — mayfer · 2026-09-02
- Supply Chain Bottlenecks: Millions of Robots Need Billions of Precision Parts — Scobleizer · 2026-09-02
- Bought a $6,279 RTX 5090 Prebuilt: Was it the right choice for local AI? — Sexyvette07 · 2026-09-02
- Hacking Feetech servos: Tuning registers for better performance — kamathsblog · 2026-09-02