100% in sim, 0/11 real robot inserts: joint team exposes benchmark inflation in GPT-6-Astra eval

jiqizhixin · x · 2026-10-29

A joint team from SenseTime, Ropedia, Nanyang Technological University, and the University of Tokyo evaluated GPT-6-Astra across 12 simulation benchmarks and 8 real-robot platforms, showing embodied scores depend heavily on what tests allow:

Core contrast: 100% on a simulation benchmark, zero successful plug insertions in 11 real-robot attempts
The study lets the model build frozen knowledge packages from permitted training demonstrations and extends execution budgets up to 10x: RoboDojo success rises from 27.6% at standard budget to 81.9% at 10x — headline scores shouldn't be read as standard-budget comparisons with published baselines
Real-robot tests use no demonstrations or additional training, exposing failures in precise contact, state verification, recovery, and preserving earlier progress
The team also prototypes a household evaluation where the clock keeps running (the figure shows the furniture placement, physics binding, and intent/duty-based task design pipeline)

Takeaway: embodied SOTA numbers must be read against what the test permits; the sim-to-real gap remains the core problem.

Original post →

100% in sim, 0/11 real robot inserts: joint team exposes benchmark inflation in GPT-6-Astra eval

More from Embodied