Hugging Face's RoboColosseum: 5,699 blind trials show AI agents rival VLA policies, no humanoid task completed
DJiafei · x · 2026-11-06
RoboColosseum: Real-World Robot Evaluation at Scale
Hugging Face and partners released RoboColosseum (arXiv:2611.02497), a distributed framework for evaluating physical intelligence in the wild: unified policy serving plus standardized protocols across 124 globally distributed sites with locally operated robots.
Design
- Open Track: blinded pairwise trials with no task-specific adaptation — 5,699 comparisons under matched local conditions
- Fine-tune Track: each VLA policy gets the same 100 demonstrations per task across 20 tasks and 4 embodiments (SO-101, Franka, Bimanual-YAM, G1 Humanoid)
- Subjects: 5 VLA policies + 3 frontier AI agents
Key findings
- Frontier AI agents rival specialized VLA policies, yet no system completed a single humanoid task
- After fine-tuning: GR00T N1.7 leads on SO-101; pi0.5 leads on Franka and Bimanual-YAM
- All systems struggle with open-world tasks; policy rankings show weak agreement across settings
Related event: Hugging Face and Partners Launch RoboColosseum Open Robot Eval Platform(75 posts)→
More from Embodied
- Chat models beat robot-specific policies in blind tests: Opus 5.5 +12.1pp, GPT-6 Astra +11.0pp — DJiafei · 2026-11-06
- Tesla Roadster Rocket up close: 10 SpaceX cold-gas thrusters, invite-only — beffjezos · 2026-11-06
- PHOS: A Raspberry Pi Robot That Gives Claude Desktop Sensors and Eyes via MCP — jfrankie · 2026-11-06
- Tesla Robotaxi crosses 4.5M autonomous miles with zero injuries and no safety monitor — XFreeze · 2026-11-06
- Google Launches Fitbit Edge: 50m Water Resistance and First User-Replaceable Battery — GoogleDesign · 2026-11-06
- Meta lets US users video chat with their Muse AI avatar ahead of wearable launch — TechCrunch AI · 2026-11-06