HandsOn Benchmark Launches, GPT-6 Astra Fails Dexterity Test

Episode 1 · HandsOn Benchmark Released and Open-Sourced for Generalist Dexterous Manipulation (2026-10-27, 5 posts)

To measure progress of general dexterous manipulation models like Natura-1, @stingning's team released and open-sourced the simulation platform and benchmark HandsOn on October 27. Results show even today's strongest policies remain far from general dexterous manipulation.

Confirmed

• The benchmark includes 66 human-verified tasks (50 core + 16 transfer) and 18,000 curated human demonstrations, roughly 56 hours of data — all collected in simulation with three synchronized views, augmented by re-rendering and prompt randomization; the dataset is split into 15,178 training and 2,822 validation demonstrations
• Evaluation has two parts: Core-50 provides demonstrations for 50 tasks for policy learning and evaluates execution; Transfer-16 contains 16 transfer tasks with no target-task demonstrations — 7 asset/relation tasks, 6 skill-composition tasks and 3 others — testing zero-demonstration transfer
• On Core-50, the best multi-task checkpoint, OpenWAM, reached only a 29.1% macro-average task success rate, with wide capability gaps: pooled across 8 training checkpoints, pick-carry-place tasks hit 34.9% success, while tool-use-on-another-object tasks were as low as 2.9%
• Fully open-sourced resources include task environments, success detectors, evaluation tools, and the PhysX demonstration dataset

Why it matters

• With the emergence of general dexterous manipulation models like Natura-1, the field lacked a unified benchmark for measuring progress toward generalizable dexterous manipulation — HandsOn fills that gap
• The 2.9% success rate on tool-use tasks shows dexterous manipulation, especially tool interaction, remains a glaring weakness in robot learning, giving future research a clear direction and a public, reproducible evaluation foundation

• HandsOn benchmark: 66 verified tasks and 18,000 human demos for dexterous manipulation — stingning · 2026-10-27
• HandsOn benchmark: 66 human-verified tasks and 18,000 curated demos for dexterous manipulation — stingning · 2026-10-27
• HandsOn splits evaluation: Core-50 execution vs Transfer-16 zero-demo transfer — stingning · 2026-10-27
• Best policy hits only 29.1% on HandsOn; tool use drops to 2.9%, transfer near zero — stingning · 2026-10-27
• HandsOn platform open-sourced: 18K human demos, tasks, eval tools for dexterous policies — stingning · 2026-10-27

Episode 2 · GPT-6 Astra Scores Only 5.2% on Zero-Shot Dexterous Manipulation Benchmark (2026-10-27, 2 posts)

The HandsOn benchmark shows GPT-6 Astra achieves only a 5.2% success rate (26 of 500 attempts) on zero-shot dexterous hand manipulation, with failure analysis of 800 rollouts revealing issues like premature triggering.

• HandsOn dexterous-manipulation benchmark: best policy 29.1%, GPT-6 Astra just 5.2% — stingning · 2026-10-27
• GPT-6 Astra controls dexterous hands zero-shot at just 5.2% success in 500 trials — stingning · 2026-10-27