One photo, 8 VLAs, only 3 correct: ERQA embodied-reasoning benchmark questioned
giffmana · x · 2026-11-22
Ville Kuosmanen flags problems with the ERQA embodied QA benchmark: one table-wiping question had its yellow and blue curves mislabeled, raising doubts about dataset quality. Testing eight VLAs on the same question, only WALL-OSS flow, MolmoAct2 and WALL-OSS 0.5 answered correctly, while OpenVLA, π0, π0.5, GR00T N1.7 and G0.5 all failed — several degenerating into garbled text. He questions whether such benchmarks meaningfully measure embodied reasoning.
More from Embodied
- Tesla FSD 15's real-time view isn't 3D rendering — it's the AI model's own synthetic perception — mark_k · 2026-11-22
- China Installed 59% of New Industrial Robots in 2026 Report, ~354,000 Units — ingliguori · 2026-11-22
- Apple Watch bricks after one shower while a $15 Keep band survives many — oran_ge · 2026-11-22
- Tesla FSD V15 visualizations go viral with detailed 3D night-time rendering — DimaZeniuk · 2026-11-22
- Tesla FSD visualization: from dots on a screen to photorealistic 3D night views in six years — XFreeze · 2026-11-22
- YQ Bionics' Y-Hand M2 packs 38 degrees of freedom as AI meets robotic manipulation — CurieuxExplorer · 2026-11-22