Why testing robot models is even harder than testing LLMs — the RoboDojo catch
binarybits · x · 2026-10-19
A long post from Understanding Robots (Kai Williams) on why evaluating robot models is harder than evaluating LLMs, plus a correction — he had assumed no top robot models would be offered via API, then Physical Intelligence shipped a beta last week.
The core: while investigating GPT-6 Astra's robotics skills on RoboDojo (built by academics from HKU, UC Berkeley, Tsinghua and others), where Astra briefly topped the leaderboard over specialized models like Physical Intelligence's, he found the story more complicated than the headline:
- RoboDojo's simulation benchmark has 42 tasks across generalization, precision, long-horizon, memory, and open tasks.
- Astra struggled with precise manipulation (e.g., inserting tubes into a rack) but aced "book smarts" tasks like arranging number tiles to form the largest number.
- The strongest specialized robot models showed the opposite: better precision, worse semantic generalization.
- Enough tasks tested semantic skills that Astra scored the highest average — masking its actual capability profile.
Conclusion: today's best robotics benchmarks are far less informative than LLM benchmarks, and headline scores mislead judgments about how fast robot capabilities are advancing.
More from Embodied
- Chinese humanoid robot shows off pole dancing skills in viral demo — LexiLove · 2026-10-20
- F-22 controls a CCA drone for the first time, Anduril and USAF hit milestone — BrettKrieger12 · 2026-10-20
- Ropedia W0 shows 10k hours of human video yields a robot learning scaling law — liuziwei7 · 2026-10-20
- Apple Sets October 28 Event, Touchscreen MacBook Pro Reportedly Unveiled — Polymarket · 2026-10-20
- Weekend DIY: two RP2040 boards give AI agents keyboard-video-mouse control of phones — hugs · 2026-10-20
- LidarVerse: one real-time LiDAR world model for reconstruction, forecasting and generation — ZGojcic · 2026-10-20