SimAudit: LLM pipeline finds 61 defects in simulated manipulation benchmark success checkers

MBZUAI · hf · 2026-11-20

MBZUAI and Peking University introduce SimAudit, an LLM-based pipeline that audits hand-written success checkers in simulated manipulation benchmarks. It identifies instruction-checker disagreement scenarios, recreates them in simulation, and verifies behavior via multi-view rendering. Across RoboTwin 2.0, RoboCasa365, and RoboDojo, 183 test cases surfaced 61 distinct defects in 49 tasks — 40 phantom successes and 21 silent failures — showing expert-demonstration validation alone is insufficient.

Original post →

More from Embodied

Embodied channel →