CI-MSE offline metric hits -0.87 Spearman correlation with real robot task success vs weak raw MSE
chris_j_paxton · x · 2026-10-16
A new arXiv paper introduces Critical Interval MSE (CI-MSE), an offline validation metric for robot manipulation policies that uses a zero-shot agent to localize task-critical segments and computes error only there, with action alignment to match rollout behavior.
Key results:
- On 6 MolmoAct2 checkpoints (fine-tuned on 44 block-sorting episodes, 20 rollouts each), CI-MSE correlated with real task success at Spearman ρ = -0.89 (p = 0.02), while raw MSE showed a weak, non-significant +0.31
- Across 27 policy checkpoints, CI-MSE achieved -0.87 vs raw MSE's -0.61
- Sensitivity analysis shows robustness across hyperparameter settings
The upshot: CI-MSE could ease the bottleneck of costly, hard-to-reproduce real-world evaluations for imitation-learning iteration.
More from Embodied
- Tesla Roadster Thruster Teardown: SpaceX COPV Tank Plus Optimus Actuator — XFreeze · 2026-10-16
- Critic Calls Figure's Terminator-Style Robot Demo Publicity Over Substance — FuSheng_0306 · 2026-10-16
- Snap Spectacles + Gemini 4 Argon demo designs LEGO builds from your brick pile — stspanho · 2026-10-16
- Waymo Announces Its Robotaxi Service Is Coming to Toronto — ChrisArchitect · 2026-10-16
- Polymarket odds of Tesla shipping Optimus this year jump 5 points to 12% after flying Roadster reveal — Polymarket · 2026-10-16
- More companies sell robot intelligence than robots themselves, as Dexmate's Vega becomes a research hub — lukas_m_ziegler · 2026-10-16