Sensorimotor Alignment: Training-Free Metric Predicts Robot Policy Success, r=0.83 on Real Robots

Sensorimotor Alignment: A Platonic Proxy for Embodied Policy Success

Yunfei Xie, Howard H. Qian, Shiyi Lan, Kaiyu Hang, Chen Wei

cs.CV, cs.LG, cs.RO

2026-10-23

S² scores alignment between vision-text features and end-effector trajectories; it tracks task success across VLAs, VLMs, WAMs and real robots (r up to 0.83) before any rollout.

What problem this solves

Robotic foundation models are still graded the slow way: run the policy closed-loop in simulation or on a real robot and count successes. That is expensive, noisy and benchmark-specific. The cheap proxies people use instead measure something else. VQA scores probe language perception; FID, SSIM and PSNR probe the pixel quality of generated video. None of them asks whether a representation is organized around the motion the robot must execute. Two visually different scenes can call for the same trajectory, and two similar-looking scenes can demand different ones.

This paper from Rice University and NVIDIA imports the Platonic Representation Hypothesis (PRH) into robotics. PRH claims that as models of different modalities scale, they converge to a shared statistical model of reality, with images and text as two projections of it. The question here: is a robot's motion trajectory another such projection, so that a more capable embodied model carries sensory representations whose geometry matches the geometry of motion?

Method

Sensorimotor Alignment (S²) requires no training; one forward pass over a reference dataset (DROID or BridgeDataV2) is enough.

• For each sampled episode, take one random frame paired with the task instruction, run it through the model's own vision and text encoders, and average-pool the tokens into a single sensory vector.
• The motor representation is the end-effector state over the next h steps, seven numbers per step (xyz position, roll/pitch/yaw, gripper), with distances treating position as vectors in R³ and orientation as rotations in SO(3).
• Sensory space uses cosine distance; motor space uses Dynamic Time Warping, so trajectories executed at different speeds still match.
• Each episode gets its k nearest neighbors in both spaces (k=10); the score is the overlap of the two neighbor sets, averaged over all episodes, reported as a centered, normalized CKNNA variant.

The intuition: if episodes the model finds similar also move similarly, the neighbor sets coincide and S² is high. A negative control shows the metric pays attention to embodiment: scoring bimanual world action models against a single-arm reference dataset drops the success correlation from +0.999 to +0.246.

Results

The correlation survives changes of reference dataset (DROID and BridgeDataV2 both near 0.7) and trajectory horizon (r>0.5 up to h=220 on DROID). On the three WAMs that generate video during evaluation, S² recovers the success ordering at r=+0.999, while SSIM (p=0.311), PSNR (p=0.345) and FID (p=0.262) are all non-significant; FID even points the wrong way, with better-looking video going along with lower success.

S² also explains four closed-loop phenomena at the representation level:

• Weak instruction grounding: after fine-tuning, success tracks vision-feature S² (+0.694) and ignores text-feature S² (+0.237); raw VLMs show the reverse (text +0.501 over vision +0.425). Fine-tuned VLAs act on what they see.
• Freezing the vision encoder cuts S² by 41% and SimplerEnv success from 50.00 to 30.21, tying end-to-end gains to alignment.
• In three WAMs, S² is nearly flat across the denoising schedule (variation within about 6%), which explains why skipping iterative video generation costs so little.
• In a 30-block video backbone, block 14 carries the highest S² in all three models, the same mid-depth range prior work identifies as semantically critical.

Why it matters

The practical payoff is cheap ranking. A VLM backbone can be scored with S² before anyone pays for VLA fine-tuning, and it beats reading VQA scores, which correlate at −0.154 with final success. Teams budgeting closed-loop rollouts get a pre-filter that leaves the expensive evaluation to a few candidates. The metric also hands one ruler to four design questions that previously had only behavioral evidence. Academically, this is the first extension of PRH into action, and the answer is clean: motion works as another projection of reality.

Limitations

The authors name three: the relationship is correlational, and optimizing S² during training has not been shown to help; models with similar S² can still land far apart in success; coverage spans few families, benchmarks and recipes, few models per comparison, and a single real-robot setting.

Sample sizes are the weakest part. Eight VLM backbones, five real-robot policies, six WAMs, with p-values hugging the 0.05 line (the WAM correlation sits at p=0.054, strictly non-significant), and the r=+0.999 figure rests on three models; three points fix a line. Absolute S² values live at different scales per family (roughly 0.015 to 0.045 for VLAs, 0.05 to 0.25 for WAMs), so comparisons only make sense within a family. The real-robot correlation is the strongest of all (0.831) but comes from one lab's unified protocol; whether it survives a change of environment is unanswered.

Terms

• PRH (Platonic Representation Hypothesis): the claim that models of different modalities converge to a shared statistical model of reality as they scale
• VLA (vision-language-action): a model that takes images and instructions and outputs robot actions directly
• WAM (world action model): an embodied model built on a video generator, decoding actions from generated latents
• End-effector: the gripper or tool at the end of a robot arm; its trajectory is what counts as action here
• DTW (Dynamic Time Warping): aligns two time series that unfold at different speeds before measuring distance
• CKNNA: a cross-modal alignment score based on the overlap of k-nearest-neighbor sets, used in the original PRH work

Source

• arXiv 2610.31670v1
• PDF
• Hugging Face

What people are saying

• Rice & NVIDIA's Sensorimotor Alignment predicts robot policy success without training (r=0.717) CSProfKGD

Related papers

• RealMe adapts 14B FLUX so eight 720p views reconstruct 2K volumetric video
• Tencent 4B VLM rejects absent objects at 67.4% while unifying boxes and mattes
• LayerEditor loses just 1% SSIM when the edit is rebuilt from its RGBA layers
• GAFU compresses generation features into 256 tokens and beats same-data SFT in 15 of 15 tests
• LaLa-World plans at 59.4 vs RAEv2's 48.0 and runs 13.9× faster
• FAPL lifts motion order from 48% to 61% and factor-edit accuracy to 58%

All paper explainers