RoboHarm benchmark and inspect-robots open-source eval framework for physical AI released
chooi_jeq · x · 2026-09-19
The author released the RoboHarm benchmark with 300 traces, plus inspect-robots, an open-source eval framework for physical AI (544 stars on GitHub). Think Inspect AI for robotics: define a benchmark once, then run any policy (LLM agent or VLA) on any embodiment — a real arm, humanoid, or simulator — with auditable logs (grader scores, LLM transcripts, full config) and first-class Rerun visualization. The project is in early development with a changing API; pin a version before depending on it.
More from Embodied
- Nano-texture iPhone Duo display makes Apple Pencil feel like writing on paper — rudrank · 2026-11-04
- NEURA launches Neuron AI, but Europe's Physical AI test is whether the stack matures together — ingliguori · 2026-11-04
- Baidu's Apollo Go wraps two-week public robotaxi test rides in Tung Chung, Hong Kong — Baidu_Inc · 2026-11-04
- imec neural probe integrates photonics directly into 130nm CMOS backend — jwt0625 · 2026-11-04
- Chris Paxton: interactive robot sim is the future but glaringly data-limited — Scobleizer · 2026-11-04
- DreamX: video diffusion models as universal latent action learners for robots — kaist-ai · 2026-11-04