One photo, 8 VLAs, only 3 correct: ERQA embodied-reasoning benchmark questioned

giffmana · x · 2026-11-22

Ville Kuosmanen flags problems with the ERQA embodied QA benchmark: one table-wiping question had its yellow and blue curves mislabeled, raising doubts about dataset quality. Testing eight VLAs on the same question, only WALL-OSS flow, MolmoAct2 and WALL-OSS 0.5 answered correctly, while OpenVLA, π0, π0.5, GR00T N1.7 and G0.5 all failed — several degenerating into garbled text. He questions whether such benchmarks meaningfully measure embodied reasoning.

Original post →

More from Embodied

Embodied channel →