PRM-as-a-Judge 1.5: A Toolkit for Robot Process Assessment
Yuyang Liu, Yanqing Shen, Ruike Chen, Jifan Zhao, Yuxuan Tian, Yichi Zhang, Tianfeng Long, Zixuan Yin, Yipu Wang, Ziheng Qin, Wenxing Tan, Yang Shi, Mingyu Cao, Runze Xiao, Ziqi Wang, Zhixin Yin, Shiwei Chu, Yi-Fan Zhang, Yao Mu, Yuheng Ji, Yihao Wang, Jun Yan, Zhongyuan Wang, Pengwei Wang, Xiaolong Zheng
cs.RO, cs.CV
2026-08-14
A CASIA-led team releases PRM-as-a-Judge 1.5, which turns robot rollout videos into dense progress curves and derives nine process metrics; pi-0.5 ranks strongest overall, sim-to-real progress drops 16.5 points on average, and model size shows no clear correlation with performance.
Robot-manipulation evaluation is dominated by binary success rates. Two failed rollouts, one collapsing at the start and one losing it at 90% done, count identically; two successes, one clean and one scraped through via hesitation and correction, also count identically. Rule-based scoring helps but cannot keep up with diverse failure modes. Evaluation should say where a model stalls, how long it hesitates, and whether it recovers.
The pipeline takes a rollout video plus task instruction, uses a process reward model to estimate per-frame task completion into a dense progress curve, then derives metrics and reports. The OPD metric suite (Outcome-Process-Diagnosis) keeps MC@25/50/75 milestone coverage, Max Progress, path-weighted progress length, cumulative regret area, and stagnation ratio from version 1.0, and adds three conditioned metrics: Failure Near-Success (how close failed rollouts came to completion), Drawdown Recovery Ratio (recovery after the largest setback), and Success Quality Score (how clean successful executions are). The default judge is Robo-Dopamine (Forward). All assessment runs on rollout videos released by the RoboDojo benchmark, 9 real-world and 16 simulation models, frozen at the 3 July 2026 leaderboard. RoboPulse++ accompanies the toolkit: 700 trajectories and 2,244 human-annotated Rising/Falling intervals for testing judge models themselves.
Specialized PRMs clearly beat general-purpose VLMs on RoboPulse++: Robo-Dopamine (Forward) reaches 0.84 Macro-F1 and 0.82 accuracy, while the best VLM configuration manages 0.50/0.66. Recognizing regression is much harder than recognizing progress: best Rising F1 hits 0.92 while best Falling F1 stops at 0.63, with low recall as the bottleneck. Of 155 sampled Falling errors, 78.1% are interaction-relation failures (dropped or misplaced objects) versus 21.9% task-order failures. On the real-world leaderboard, pi-0.5 tops MC@25 (85.29), MC@50 (60.59), SR (17.06), and FNS (45.39); Spirit v1.5 never succeeds in the measured rollouts. Five headline findings: VLAs generally outperform WAMs (100%/80%/60% VLA share in Sim Top-3/5/10 versus 0%/20%/40% for WAMs); larger models do not guarantee better performance, with 4B pi-0.5 beating 5-6B WAMs; pi-0.5 is the strongest overall; Precision tasks are the best-performing category while open-vocabulary tasks are hardest; and simulation does not represent real-world performance, with SR rank correlation of 0.58, MP of only 0.18, all nine shared models lower in real than sim, and an average Max Progress drop of 16.5 points. Progress curves also recover conventional success rates with 1.3-1.6 point MAE and model-level rank correlations of 0.88 (sim) and 0.96 (real), so nothing is lost versus SR. On a single H100, batching cuts Robo-Dopamine-4B runtime on 2.1 hours of video from 29.3 to 7.8 minutes.
For embodied-AI teams this turns "where does the model fail" from video-watching guesswork into curve readouts that point at data and training fixes. The mismatch between SR and process-metric rankings means success-rate-only selection systematically misleads. The 0.18 sim-real MP correlation is a bucket of cold water: topping a simulator says little about deployment, and precise-alignment tasks (Classify objects, Hang mugs, Insert tubes) degrade most. RoboPulse++ makes the judge itself measurable, and quantifies the weak Falling recognition that limits drawdown-based metrics.
Every conclusion rests on rollout videos from a single third-party benchmark. The default judge being Robo-Dopamine creates a same-family bias the paper cannot rule out by itself, and with Falling F1 at 0.63, regression-sensitive metrics (CRA, DRR) carry real noise. SQS is not compared on the real-world side because most models' SR is too low for a reliable estimate. The nine shared models bound the sim-real analysis. Whether process-based evaluation will invite optimization against progress curves goes undiscussed, and the author list overlaps with several evaluated teams.