Benchmarking LLMs on robots: GPT-6 Astra scores 36% vs Claude Fable's 15%
k7agar · x · 2026-09-18
Robotics firm Pantograph benchmarked GPT-6 Astra vs Claude Fable 5.1 on their Pandroid bimanual mobile robot with identical prompts, tool sets, and budgets: 8 simple manipulation tasks, 10 attempts per model per task, 160 rollouts total, capped at 20 minutes or 96 turns, with blind, randomly interleaved grading.
Results: Astra completed 36% of attempts vs Fable's 15%, statistically significant (p≈0.001), with most of the gap coming from two tasks. Astra on low effort takes faster turns with fewer tokens. On easy tasks like placing a ball in a bowl Astra scored 10/10, while neither model could get a beanbag out of a cabinet drawer.
Takeaway: both frontier models remain far from human performance — teleoperators easily score 100% in much less time.
Related event: GPT-6 Astra Doubles Claude Fable's Success Rate in Robot Control Benchmark(2 posts)→
More from Embodied
- Autonomous's Harness Pro uses an ESP32-P4, not a mini PC, to monitor AI coding agents — dee_hw · 2026-11-02
- KAIST's QCP lets robot policies control execution quality in real time — kaist-ai · 2026-11-02
- ANYbotics begins customer deliveries of ANYmal X, first legged robot certified for explosive atmospheres — lukas_m_ziegler · 2026-11-02
- RAM prices tripled, so Tesla cut a third of memory from Optimus's brain — aakashgupta · 2026-11-02
- Too lazy on weekends? Watch this robot tidy up a desktop — flngr · 2026-11-02
- Wearing Whoop and Apple Watch on a single dual-slot band — tinyfool · 2026-11-02