MBZUAI's Ego2Act Benchmark Shows Video Models Skip Steps and Break Physical Plausibility
MBZUAI · hf · 2026-10-03
MBZUAI introduces Ego2Act, a benchmark for goal-directed egocentric video generation testing video models as world simulators:
- Scale: 2,640 videos across 110 real-world everyday tasks with varying clutter and multi-step complexity; given a scene image and high-level goal, models must simulate hands manipulating objects to complete the task.
- Automated eval: Ego2ActJudge, a reference-free judging pipeline, aligns with human consensus better than baselines.
- Key findings: generated simulations often skip or partially execute steps, leaving later steps missing dependent states and goals unfulfilled; models consistently fail at fine-grained physics, complex manipulation, and persistent world modeling.
The authors position Ego2Act as a rigorous testbed for physically plausible, goal-directed simulation.
More from Embodied
- ETH's Anymal robot completes 10km autonomous alpine hike with VLM-guided ROAM system — ChongZzZhang · 2026-11-19
- iFixit tears down Apple HomeView: lots of glue and 'endless frustration' — Resident-Fall4351 · 2026-11-19
- Removing human video drops robot OOD success from 60% to 7%: generalization comes from human video — gui_penedo · 2026-11-19
- PSYONIC's prosthetic Ability Hand now used by 90+ robotics orgs, raising Series A — DhowCIO · 2026-11-19
- Simulator misspecification breeds overconfidence: sim-to-real study on SPEED+ — Nikolay Dolgov · 2026-11-19
- Doppel world model renders the robot body and beats action-only baselines across 8 embodiments — mangahomanga · 2026-11-19