Team spends $5K benchmarking GPT, Claude, Gemini on LIBERO-10 — best success rate just 44%
YuXiang_IRVL · x · 2026-10-22
A robotics team spent $5K evaluating 6 frontier models from GPT (Astra/Sol/Luna), Claude, and Gemini on the LIBERO-10 manipulation benchmark, releasing all baselines for reuse.
- All runs zero-shot, no task-specific training; LLMs predict end-effector poses
- 3 reasoning levels per model, 18 configurations, 1,800 episodes total
- Best success rate: only 44%; more reasoning helps but API cost varies widely across models
- Includes a success-rate vs. API-cost comparison
Takeaway: impressive demo videos aside, LLM reasoning-to-action on standardized robot benchmarks still has a long way to go.
More from Embodied
- iPhone Duo footage shows noticeable HDR artifacts resembling halation filter — flngr · 2026-10-22
- Harness Pro: a $290 desk screen for watching and directing every AI agent — dee_hw · 2026-10-22
- COWBOY: sim-to-real framework gives mobile manipulators zero-shot whole-body grasping in the wild — rsalakhu · 2026-10-22
- 8-year-old builds fully offline voice-controlled AI storyteller on an $8 ESP32 — JarnoDuursma · 2026-10-22
- Nigerian teen builds AI smart glasses that let blind people walk without a cane — JarnoDuursma · 2026-10-22
- Baidu's Apollo Go robotaxis hit London streets with Lyft's FREENOW — Baidu_Inc · 2026-10-22