LeRobot Unifies Robot Evaluation CLI
RemiCadene · x · 2026-07-15
LeRobot 0.6.0 introduces a unified lerobot-eval CLI to evaluate various VLA robot benchmarks, eliminating the hassle of juggling multiple repositories and environments.
This release adds 6 simulation benchmarks, each equipped with dedicated documentation, Docker images, and CI-verified SmolVLA baselines:
- LIBERO-plus: 10,000 perturbation variants across 7 dimensions to observe policy failure conditions
- RoboTwin 2.0: 50 bimanual tasks based on SAPIEN, with 100,000+ trainable trajectories on the Hub
- RoboCasa365: 365 kitchen tasks across 2,500 procedurally generated kitchens
- RoboCerebra: Long-horizon tasks chaining 3–6 sub-goals
- RoboMME: Memory tests including counting, tracking, and imitation
- VLABench: Knowledge and reasoning tasks, from physics to end-to-end "making coffee"
Combined with LIBERO, Meta-World, and NVIDIA IsaacLab-Arena, there are now 9 benchmark families runnable under a single toolset.
More from Embodied
- LIBERO-MAX: Robot Policies Drop 11–26 Points When the World Changes Mid-Task — drmapavone · 2026-10-27
- Photos from ROSCon UK: BBC's robotic whale, da Vinci surgical robot, PR2 and Baxter — kamathsblog · 2026-10-27
- Agility's Digit robot to expand into two more GXO US warehouses from early 2027 — jonstephens85 · 2026-10-27
- Real2Sem2Real turns one phone video into 2,000+ robot demos, matching 150 teleop demos — Scobleizer · 2026-10-27
- Wayve CEO argues end-to-end deep learning should power driving, safety and world models — alexgkendall · 2026-10-27
- Mundo turns a few room photos into an explorable, simulatable 3D world — at 1.5 FPS — mark_k · 2026-10-27