Embodied Agent Arena: Closed-Weight VLMs Lead All Five Robot Task Domains, Open-Weight Models Lag
Haojian Huang · hf · 2026-11-04
Researchers from HKUST(GZ), CUHK and Knowin AI introduce Embodied Agent Arena, a benchmark testing whether frontier vision-language models can act as robot generalists.
- Covers five domains—geometry, spatial reasoning, affordance, task planning, and manipulation—with 1,000 cases combining 32 established sources and GeoProbe, a new 168-case benchmark for camera, motion, depth, and scale estimation
- Evaluates 10 VLM agents via a minimal harness that leaves task decisions to the model
- Closed-weight models lead all five domains, while open-weight VLMs show useful spatial reasoning but substantially lower planning and manipulation success
- Failure analysis reveals similar average geometric errors conceal different scale/shape distortions, and valid region estimates don't necessarily identify usable action points
More from Embodied
- 22 Creative SAM 3.1 Use Cases From Meta's AI Developer Hackathon, From Wildfire Detection to Document Reassembly — Vjeux · 2026-11-04
- Ex-OpenAI robotics intern breaks down viral Astra robot-painting demo on Cerebras podcast — AccBalanced · 2026-11-04
- PrimeBot's PrimeSim: a generalist world model simulator trained on cross-embodiment demos — Travis Davies · 2026-11-04
- Physical AI Hackathon in SF Nov 7–8 features Dream-1 world action model and robot arms — Scobleizer · 2026-11-04
- Figure robot office demo looks like a sci-fi movie, says CEO Brett Adcock — adcock_brett · 2026-11-04
- 'Beyond Validation Loss': best pretraining checkpoint may hurt downstream RL fine-tuning — SongShuran · 2026-11-04