RynnValue: Scaling Robotic Value Foundation Models with Temporal Distance
Dongchi Huang, Hongyin Zhang, Bohan Hou, Siteng Huang, Zhian Su, Hang Guo, Tong Lu, Zhaofeng Xu, Jiahao Tang, Jianfei Yang, Donglin Wang, Peixi Peng, Mingxiu Chen, Deli Zhao, Xin Li
cs.RO, cs.CV, cs.LG
2026-08-11
RynnValue supervises robot value learning with temporal distance, not preferences; on 7,000 hrs it beats the supervised SOTA and lifts online policy success from 52.5% to 72.5%.
Generalist robot policies increasingly rely on reinforcement learning, but the bottleneck in that loop is reward supervision, not policy capacity. Hand-designed rewards barely generalize to open-ended tasks, and sparse success signals give little guidance over long horizons. Existing general-purpose reward models tie supervision to task-internal anchors: preferences (pairwise trajectory comparisons), reference demonstrations, or local state comparisons, all of which bind to particular trajectories or comparison sets and resist reuse across heterogeneous data. The common fallback is progress normalized to [0,1], but that is an intra-trajectory coordinate rather than a goal-conditioned cost-to-go, misaligned with the standard notion of value in control and hard to keep consistent across durations, embodiments, and task structures.
RynnValue reframes the problem: from a reward model scoring trajectory-level anchors to a value foundation model predicting goal-conditioned cost-to-go, with temporal distance as its scaling target. Temporal distance is the directed temporal cost from an observation to the language-specified goal; under a minimum-time objective it is the hitting-time cost-to-go. Crucially, temporal-distance labels can be computed directly from timestamps once a completion cutoff is set, so RynnValue trains on over 7,000 hours and roughly 3 million instruction-conditioned clips with no preference or progress annotations. The clips come from ten datasets (AgiBot, EgoDex, Open X-Embodiment, and others), unified under one temporal-distance interface after subtask segmentation and cutoff relabeling.
Piling on data alone does not guarantee reliable temporal-value learning. In a multi-frame setting the model takes shortcuts: exploiting fixed sampling intervals, the presentation order of observations, or extrapolating from other exposed value queries instead of reading the visual evidence. RynnValue suppresses these with several designs together:
The model is built on RynnBrain with eight repeated query tokens per prediction; two distributional heads (absolute and relative) regress over 256 symlog bins with two-hot targets, while a language branch produces video description and match/success judgments.
Temporal distance is then converted to a dense reward through potential-based shaping, using the negated prediction as the potential and preserving the temporal scale rather than normalizing to [0,1].
On the RBM-EVAL-OOD trajectory-ranking benchmark (976 trajectories across six out-of-distribution datasets, Kendall's taua), RynnValue-8B averages 0.675 and the 4B variant 0.670, surpassing Robometer (0.655), which is trained with both preference and progress supervision, and more than doubling the progress-only ablation (0.292). Among methods trained without preference supervision, RynnValue is best on all six datasets, lifting the strongest preference-free average from 0.502 to above 0.670. Its instruction-trajectory confusion matrix yields a diagonal margin of 0.79, above the strongest baseline's 0.67, showing the estimates are anchored to the language goal rather than generic visual progress.
Ablations attribute each component clearly: removing temporal-order shuffling drops the score to 0.189 (the largest loss), uniform sampling to 0.379, value-isolation to 0.482, language supervision to 0.537, and the relative head to 0.627. A scaling study shows that adding task diversity reduces error monotonically while adding episodes within a task saturates quickly: diversity beats quantity.
In real-world reinforcement learning (Franka dual-arm, four tasks, 20 trials each, zero-shot with no target-domain fine-tuning), online RL averages 72.5% success for RynnValue versus 52.5% for Robometer and 48.8% for sparse rewards; offline RL (IQL on pi-0.5) reaches 82.5% versus 63.8% for Robometer and 23.8% for SFT, and it solves Box-in-Drawer and bimanual box transfer, which SFT never solves at all.
RynnValue swaps the most constricting part of robot learning, reward annotation, from costly human preferences to nearly free timestamps, and shows temporal distance is a scalable supervision target that unifies embodiments, viewpoints, and data sources. Weights, code, and a project page are all open, and the model drops in as a reward interface for policy training.
The authors concede that the model currently estimates temporal distance from a short window of sampled observations, leaving longer horizons and streaming inference to future work; the objective assumes an approximately minimum-time target and ignores task-specific costs such as energy, safety, or precision; and they plan to extend to dexterous hands and mobile manipulation, where navigation and interaction must be jointly grounded. Two more from reading: online gains are limited on precision tasks like Box-in-Drawer, because the reward sees only third-person RGB and visually similar poses can correspond to very different grasp stability, making intermediate rewards hard to calibrate; and the small 4B-to-8B gain both shows the recipe is not scale-bound and raises the question of whether scaling further is worth it. The shaping also retains a human-annotated sparse success terminal, so the setup is not fully autonomous.