Robot Leaderboard Under Fire: 5 Trials Per Task, Authors Commit to 15
YuXiang_IRVL · x · 2026-09-21
A real-robot leaderboard sparked a methodology debate. Critic Will argued 5 trials per task is a "highlight reel" lacking confidence intervals, cycle time, jam recovery, and intervention data. UCSD's YuXiang clarified the setup (5 trials × 10 tasks), committed to 15 trials per task, and noted 6-8 hours per policy across 7 policies already evaluated — underscoring the cost of rigorous real-world evaluation.
More from Embodied
- UBTECH sells 400 U1 Pro humanoids to Yongda Auto in ~$10M deal — CyberRobooo · 2026-11-12
- Baidu World 2026 to showcase Apollo Go robotaxi rides and Apollo World physical AI model — Baidu_Inc · 2026-11-12
- NVIDIA's open-source Alpamayo 2 Edge VLA driving model lands in Autoware — drmapavone · 2026-11-12
- A closer look at Astra's robot gripper design (and a joke at its expense) — _Stocko_ · 2026-11-12
- Robot startup passes first presale milestone with 10+ units claimed — R4nsack · 2026-11-12
- Runway launches Praxis-1, a 27B MoE world action model, with a 73ms Fast variant — ChongZzZhang · 2026-11-12