XDOF launches real-robot leaderboard benchmarking frontier models on 16 physical tasks
pabbeel · x · 2026-10-22
XDOF announced its Robotics Leaderboard, rigorously evaluating frontier models on real robots to address the gap between flashy zero-shot demos and actual physical capability.
- First benchmark, Ember-0: 16 shared tasks, 30 reps each, scenes reset and randomized; models act fully autonomously except for hardware/safety intervention
- Scoring: equal-weighted average task success across 16 tasks, broken down by capability tags and action types
- Uncertainty estimated via 95% confidence intervals from resampling recorded runs; trophies mark observed leads, not statistical significance
- Covers 4 models so far, updated with every new evaluation; human scores get separate review
UC Berkeley professor Pieter Abbeel amplified the launch, framing it as targeted measurement of where frontier models still fall short physically.
More from Embodied
- Samsung tests humanoid robot for 24/7 factory shifts, weighing CES 2027 debut — CyberRobooo · 2026-10-22
- A week with the dual-screen iPhone Duo: gestures, apps and the camera compromise — BenBajarin · 2026-10-22
- First public demo of automated T-shirt production line in North Carolina — MatthewChang · 2026-10-22
- DeepMind's Gemini Argon Compresses Turing Machines for Busy Beaver, Goldbach, Riemann in Lean 4 — robertarail · 2026-10-22
- Unitree unveils H2-X kung fu guardian robot with ~8h endurance that can lift a car — LinusEkenstam · 2026-10-22
- Sharpa's D01 humanoid: 21-DoF tactile hand, 0.2mm precision and a haptic data glove — arnie_hacker · 2026-10-22