Chat models beat robot-specific policies in blind tests: Opus 5.5 +12.1pp, GPT-6 Astra +11.0pp
DJiafei · x · 2026-11-06
A new RoboColosseum paper ran blind head-to-head comparisons across 124 real-world sites and found general chat models now competitive with robot-specific policies — with no fine-tuning and no demonstrations.
- Claude Opus 5.5 led robot VLAs by +12.1 percentage points in mean task progress; GPT-6 Astra by +11.0
- All agents averaged +5.3, while Grok 4.7 underperformed at -7.5
- Across 901 agent-vs-VLA comparisons: agent made more progress 394 times, VLA 265, ties 242
- On the Unitree G1 humanoid, none of the robot-specific models finished a single task
Takeaway: frontier general models can already replace some specialized robot policies out of the box.
More from Embodied
- RoboColosseum launches open robot evals: 100+ robots, 10,000 free eval hours — DJiafei · 2026-11-06
- DIY walkie-talkies and charms for talking to AI agents are popping up on Reddit — CandyMonty · 2026-11-06
- Tesla Roadster Rocket up close: 10 SpaceX cold-gas thrusters, invite-only — beffjezos · 2026-11-06
- PHOS: A Raspberry Pi Robot That Gives Claude Desktop Sensors and Eyes via MCP — jfrankie · 2026-11-06
- Tesla Robotaxi crosses 4.5M autonomous miles with zero injuries and no safety monitor — XFreeze · 2026-11-06
- Google Launches Fitbit Edge: 50m Water Resistance and First User-Replaceable Battery — GoogleDesign · 2026-11-06