SimAudit: LLM pipeline finds 61 defects in simulated manipulation benchmark success checkers
MBZUAI · hf · 2026-11-20
MBZUAI and Peking University introduce SimAudit, an LLM-based pipeline that audits hand-written success checkers in simulated manipulation benchmarks. It identifies instruction-checker disagreement scenarios, recreates them in simulation, and verifies behavior via multi-view rendering. Across RoboTwin 2.0, RoboCasa365, and RoboDojo, 183 test cases surfaced 61 distinct defects in 49 tasks — 40 phantom successes and 21 silent failures — showing expert-demonstration validation alone is insufficient.
More from Embodied
- CoRL 2026 best paper SuperMimic accused of repackaging 2019-era work with slick marketing — animesh_garg · 2026-11-21
- Perceptron Releases Ready-to-Run Checkpoints of Its 36B Robot Model Isaac 0.5 — rohanpaul_ai · 2026-11-21
- Fei-Fei Li: World Models Go Beyond Games; Robotics Market Cap Is "Truly Unbounded" — StanfordHAI · 2026-11-21
- DeepMind's Antigravity Robotics Plugin: 2 Prompts, ~20 Minutes to Command a Real FR3 Duo Arm — jackyliang42 · 2026-11-21
- Robotics startup Hummingbird hosts first Istanbul event, hiring engineers — animesh_garg · 2026-11-21
- OpenAI NYC hardware hackathon: Corner Walk turns your GPS-tracked stroll into a retro game — OpenAIDevs · 2026-11-21