Open Tool Reveals Pixel Metrics Fail to Rank World Models on Real Robot Video
georgia_bucea · reddit · 2026-08-14
The author developed an open-source tool called Worldproof to diagnose where world models (models predicting future frames) break down, leading to a counterintuitive and crucial finding.
Core Finding: Pixel Metrics Lack Discriminative Power
- On real SO-101 robot arm recordings, standard pixel metrics like SSIM and PSNR cannot effectively rank models.
- This happens because prediction errors do not grow significantly with the horizon. If predicting 6 steps ahead is no harder than 1 step, all models appear similarly mediocre.
The Usable Evaluation Window
- The author measured 48 steps on the DROID dataset and identified three regimes:
- Steps 1-3: Near-perfect ties across all models.
- Steps 4-24: Steep monotonic decline; the only stretch where models are actually separable.
- Step 28+: Prediction fully decorrelated, and all models tie again at the bottom.
Methodology & Advice
- When evaluating world models, researchers must measure their own usable window based on frame rate and task speed, rather than blindly inheriting default horizons from papers.
More from Embodied
- Tesla Unveils 'Flying Roadster' Reportedly Equipped with Cold Gas Thrusters — nptacek · 2026-10-16
- AutoAgent0: VLM-powered agentic runtime lifts long-tail driving success by 28.3pp — yining_hong · 2026-10-16
- CarryOn: dual-loop harness adds personalization and safe recovery to frozen VLA robots — Zixuan Wang · 2026-10-16
- FoveaVLA replaces wrist cameras with learned foveation, hitting 70.8% success — Haoyu_Xiong_ · 2026-10-16
- Musk teases two-way Grok x Optimus integration: Grok in the bot, bot in Grok — beffjezos · 2026-10-16
- SimVTA uses simulator-rendered video to post-train driving world action models with RL — OpenDriveLab · 2026-10-16