Multimodal Embodied Agents: A Unified Capability-centric Perspective from Computer-Use to Robot-Use
Yanzhe Chen, Ziyi Yang, Jifeng Zhu, Qiming Huang, Ruihe An, Peiyao Xu, Hesen Yang, Runda Liu, Chang Gong, Zhijun Cao, Zechen Bai, Wenzheng Zeng, Yiqi Lin, Guoqiang Liang, Kevin Yuchen Ma, Kevin Qinghong Lin, Mike Zheng Shou
cs.RO, cs.AI, cs.CV, cs.MA, cs.MM
2026-10-22
NUS survey proposes PAPAV, five capabilities that put computer-use agents and robots on one axis; across 64 benchmarks, only 5 explicitly test Anticipate and 3 test Verify.
Computer-use agents and robotic systems grew up as separate literatures, and their surveys reflect it: GUI-agent surveys foreground computational architecture and digital interaction, embodied-AI surveys foreground physical tasks and execution. The authors tabulate ten related surveys (Table 1): six cover both computer-use and robot-use, four analyze benchmarks, and none compares the two settings as one capability structure.
That gap is widening. Foundation models such as π0 and Gemini Robotics, plus world models and broader robot datasets, now connect capabilities that used to be developed separately. Strong components do not assemble themselves into a coherent physical agent; coherence depends on coordinating the full loop as the environment changes, and that needs a unit of comparison stable across both settings. The central question is blunt: what actually changes when a multimodal agent moves from operating software to operating a body?
The answer is PAPAV, five functional roles in a recurring task loop:
Two design choices carry the framework. First, roles are defined by function, not architecture: one model can serve several roles, and one role can be spread across models and tools. A click, a skill call, and an arm trajectory differ in form; PAPAV asks only what each contributes to the same loop. Second, the task step is the common time scale: a click and a control tick are not comparable, a task step is. Relative to the usual observe-decide-act loop, PAPAV splits Anticipate out of Plan (estimate consequences before choosing a course) and Verify out of observation (test whether evidence supports the intended outcome, rather than logging another observation).
What embodiment changes is condensed into five constraints: partial observability, limited simulatability, irreversibility, real-time coupling, and unverifiable outcomes. Together they leave fewer choices fixed in advance and less room to reverse errors, pushing more decisions into the interaction itself. A system qualifies as an MMEA (multimodal embodied agent) only if task state persists across rounds of physical execution, each claimed capability produces an output that can influence a later task-level decision, and decisions account for the feasibility and timing of the deployed body. Missing evidence marks a system indeterminate rather than out.
The corpus is 64 benchmarks: 10 MMEA, 23 MMA (multimodal agents), 31 RS (robotic systems). Each is labeled per capability on three levels: Explicit (a dedicated task, metric, or controlled comparison), Implicit (required, but visible only through end-to-end performance), and Not Evaluated. Guardrails against inflation are stated up front: path and motion planning do not count as task-level Plan, checks by an external evaluator do not count as the agent's Verify, and multimodal input alone is not Perceive evaluation.
| Statistic | Count |
| Benchmarks explicitly evaluating Anticipate | 5 / 64 |
| Benchmarks explicitly evaluating Verify | 3 / 64 |
| RS benchmarks explicitly evaluating Act | 29 / 31 |
| RS benchmarks explicitly evaluating task-level Plan or Verify | 0 / 31 |
| MMEA benchmarks explicitly evaluating Act | 10 / 10 |
The pattern is lopsided. Act is measured almost everywhere; prediction and outcome judgment stay hidden behind task success. All ten MMEA benchmarks explicitly test Act, Anticipate is Implicit in every one, and only RoboCerebra and AgenticLab test Verify. On the MMA side, explicit Anticipate lives in the world-model cluster (WorldModelBench, PBench, HarnessEval-W); artifact benchmarks such as SWE-bench test outputs externally, so prediction and revision go unscored.
The framework also traces capability transfer across the boundary. ShowUI-π borrows a flow-matching action expert from robot policies to generate GUI drags as closed-loop continuous trajectories from streaming screenshots, instead of a pair of coordinates. Self-Grounded Verification elicits task-completion criteria independently of the candidate trajectory, and the same mechanism is demonstrated across web agents, GUI agents, and robotic manipulation. The mechanisms travel; what changes is the evidence and the constraints.
For benchmark builders this is an audit: without separate metrics for Anticipate and Verify, a leaderboard cannot tell executing an action from achieving the goal, and failures cannot be attributed to perception, planning, or judgment. The paper also punctures a common illusion: phase-level progress computed by an external evaluator does not show that the agent itself can organize a task or recognize an intermediate failure. For roboticists it documents a concrete route for importing machinery from computer-use, with ShowUI-π as a working example. For newcomers, the capability-organized literature map plus the 64-benchmark labeling table (with an accompanying Awesome repo) is genuine navigation. The honest framing: this is taxonomy and measurement work, no new model and no leaderboard climb, and its value depends on whether the community adopts the vocabulary.
The three-level labels are manual coding by the authors' team; the protocol states boundary rules, but the paper does not report inter-annotator agreement, so single-team bias cannot be ruled out. The corpus was last updated 8 September 2026 and selection involved judgment calls; a different team might curate a different set. Fields like qualified belief and the bidirectional execution record are analytical criteria, as the authors state plainly, not things current systems report, so almost no deployed system fills the template today. The five constraints answer what changes qualitatively; no attempt is made to quantify their relative weight. And the framework itself is untested as a predictor: nothing in the paper checks whether PAPAV-based analysis forecasts which systems fail where.