SimAudit Uncovers 61 Checker Defects in Three Robot Benchmarks; Repairs Flip Up to 9.1% of Rollouts

Phantom Success, Silent Failure: Auditing the Success Checkers of Simulated Manipulation Benchmarks

Zirui Song, Jiadong Zhang, Zheng Lu, Xiaosong Ma, Xiaojun Chang, Xiuying Chen

cs.CV, cs.RO

2026-11-19

An LLM pipeline audits the success checkers of three robot manipulation benchmarks and confirms 61 defects; repaired checkers flip up to 9.1% of a policy's rollouts.

What problem this solves

RoboTwin 2.0, RoboCasa365 and RoboDojo are where manipulation policies get ranked, and every episode in them is scored by a hand-written success checker that reads simulator state. The leaderboard success rate is just the aggregate of those verdicts. Benchmark authors validate a checker with scripted expert demonstrations, keeping only seeds the expert passes, which shows the checker accepts one way of doing the task. Learned policies stop early or finish by another route and reach states the expert never visited. There a checker can credit an execution that never finished the task (a phantom success) or reject one that satisfies the instruction (a silent failure). The opening examples: a bottle lifted 3 cm off the table without being shaken counts as success, while a robot that picks up one bottle with each arm, as instructed, fails because the checker also demands both bottles sit at the table centre.

Earlier audits examined task design, shortcuts and physics-engine artifacts, but they measured effects with the benchmarks' own checkers, taking the checker itself as correct. This paper tests that assumption directly.

Method

SimAudit is an LLM pipeline in four stages.

The repaired version, the Verified checker, targets the three places a verdict can go wrong. It measures physical state directly (force on the object, not whether the open command was sent), matches conditions to the instruction (adds missing ones, drops unstated ones), and requires success to persist for one second so a stack that is already toppling does not pass. Two gates validate a repair: all 1,233 expert episodes still pass, and the rendered defect states serve as a regression suite.

Results

Across 142 tasks the agent proposed 568 candidate states; 183 returned the predicted verdict; visual review cut about two thirds, leaving 61 distinct defects over 49 tasks.

BenchmarkDefectsPhantomSilentDominant cause
RoboTwin 2.0312011proxy signal (15)
RoboCasa36518171loose geometry (8), missing condition (7)
RoboDojo1239unstated requirement (9)

Each benchmark fails differently. RoboTwin 2.0 checkers read the commanded gripper value instead of the fingers, so a placement is credited while the object is still in the grip. RoboCasa365 over-credits: a straw lying across the rim counts as inside the cup. RoboDojo under-credits: the digit 8 is rejected after a half turn that nothing in the instruction mentions. The claim that simulated success rates are inflated holds for RoboCasa365, holds in part for RoboTwin 2.0, and reverses for RoboDojo.

Precision was checked by two independent annotators, who confirmed all 61 defects from the rendered frames; a random sample of 50 rejected candidates held no missed defects.

Rescoring released policies on the same rollouts with the Verified checker:

PolicyOriginalVerifiedFlipped episodes
π0.5 @ RoboTwin 2.046.2%42.3%5.6%
GR00T N1.5 (30% data) @ RoboCasa36537.7%33.8%5.4%
VPP @ RoboDojo30.0%39.1%9.1%

No ranking changed on any benchmark, but the gaps moved: VPP closed to within about a point of Rex-M1 on RoboDojo, and the two GR00T N1.5 variants drifted further apart. Per task, the damage concentrates. On RoboTwin 2.0's click bell, π0.5 keeps none of its credited successes because the gripper stops above the bell; X-VLA falls from 95% to 20% on the same task while LingBot-VA is untouched. On move stapler pad, Evo-1 goes from zero credit to succeeding in most episodes, held back only by a 5-degree yaw requirement absent from the instruction. A leaderboard that averages over tasks reports none of this.

Why it matters

Anyone comparing policies on these benchmarks gets a scale error with a known direction: RoboCasa365 over-credits, RoboDojo likely under-credits, and the two errors point opposite ways, which matters before comparing numbers across benchmarks. SimAudit, the repaired checkers and the rendered counterexamples are released, so rescoring is immediate. The recipe travels too: an LLM proposing counterexamples from code, staging them in the original environment, and an independent VLM judging only from renders applies to any benchmark scored by code, the same move UTBoost made for SWE-bench. The weight is honest: no ranking changed here. This fixes the ruler, not the order of the race.

Limitations

No recall figure exists: two thirds of candidates were cut at review and nothing measures the defects the search never proposed, and anything judgeable only from motion or force rather than renders is out of scope by construction. The rescoring protocol is asymmetric, since the original checker ends an episode at its first success verdict while re-evaluation lets execution continue, so some recovered silent failures may include an extra-time effect. The Verified checkers are themselves repairs with a hand-picked one-second persistence rule, validated against expert episodes and 61 regression states rather than an audit of equal rigour. Coverage is three benchmarks; VLABench, flagged by an earlier manual audit, is out of scope.

Terms

Source

Related papers

All paper explainers