VLA-Replica: low-cost real-world robot benchmark, MolmoAct2 beats π0.5 by 10% OOD
YuXiang_IRVL · x · 2026-09-16
- UT Dallas's Intelligent Robotics and Vision Lab released VLA-Replica, a low-cost, easily reproducible real-world benchmark for evaluating VLA models, presented at the RSS 2026 Data-Centric Robotics Workshop.
- Built from off-the-shelf parts (SO-101 follower arm, light box, cameras), the setup can be assembled in about an hour by someone with no prior experience and replicated across labs, with consistent in-distribution and OOD evaluation protocols.
- First result: MolmoAct2, fine-tuned with only 50 demonstrations per task on the SO-101 setup, outperforms π0.5 on OOD tasks by 10%.
- The team plans to keep benchmarking frontier models for real-world manipulation, with GPT-6 Astra next; paper, code, scenes, leaderboard, and training data are open.
More from Embodied
- Wearing Whoop and Apple Watch on a single dual-slot band — tinyfool · 2026-11-02
- REA: async brain-cerebellum runtime for robots where policies never wait on the LLM — Pengfei Zhou · 2026-11-02
- Robot Combat Sports Coming to Japan with a Permanent Arena in Tokyo — cixliv · 2026-11-02
- REK robot fighting in the cage: humanoid bot in boxing gloves draws a crowd — broodsugar · 2026-11-02
- iPhone Duo: a folding iPhone mod impresses, but its hinge raises crackgate concerns — rudrank · 2026-11-02
- InflatableBots: University of Calgary's inflatable robots bring dynamic haptics to large-scale VR — lukas_m_ziegler · 2026-11-02