STEAM: A New Robot Offline RL Framework

青稞AI · wechat · 2026-07-20

This WeChat article introduces STEAM, a frame-wise advantage estimation framework for real-world robot learning proposed by Prof. Yu Chao's team at Tsinghua and Zhengxing Innovation, requiring no manual frame-by-frame annotation.

Core Idea

To address the common mixture of pauses, corrections, failures, and suboptimal actions in real robot data, STEAM uses normalized temporal offsets between frame pairs in expert trajectories as a self-supervised signal.
By training multiple temporal-offset predictors, the results are converted into advantage scores measuring local task progress.
To handle mixed-quality rollouts, the method adopts the minimum advantage value from an ensemble model as a conservative score, more robustly identifying effective progression, failures, and recoveries.

Experimental Results

In real robot tasks such as dual-arm towel folding, chip checkout, cola restocking, and single-arm grasping and placing, STEAM effectively identifies low-quality segments.
Combined with CFGRL, performance improvements reach 59% / 54.3% / 23% / 16.2% over baselines.

Additional Information

The article also previews a live stream on July 21, where STEAM first author Liu Zhihao will explain the method, compare existing approaches, discuss the framework, and present real robot experiments.

Original post →

STEAM: A New Robot Offline RL Framework

More from Embodied