Progress Reward Modeling for Robotic Learning: A Comprehensive Survey
Jianshu Zhang, Keliang Wu, Haoran Lu, Anbang Liu, Ce Zhang, Weijie Yin, Chengxuan Qian, Xiyuan Yang, Zhenyu Pan, Guo Ye, Han Liu
cs.RO, cs.CL
2026-07-23
Robot policies learn poorly from sparse success-or-failure signals. This survey unifies the scattered 'progress reward' literature under a three-part framework: interface, methods, and evaluation.
Most robot learning trains on a binary success signal handed out at the end of a task: grasp succeeded or it did not. That signal answers only "is it done?" It cannot say whether the current step is making progress, treading water, or undoing work already done. In long-horizon tasks successful outcomes are rare and feedback arrives late, so the policy has little to learn from mid-execution.
Progress rewards try to close that gap by giving feedback continuously as the task runs. The trouble is that the field has grown into a tangle of methods with different inputs, goal specifications, output forms, supervision sources, and evaluation protocols, so nobody can compare them or tell what each result actually validates. This survey builds one shared framework for all of it.
The survey organizes progress reward modeling into three layers, working from the outside in.
The first layer is the interface, treating the progress model as a task-conditioned black box and asking only what it takes in and what it emits. Inputs split into current-state representation (single frame, temporal window, before-after comparison, simulator state access), goal specification (language, goal image, programmatic structure), and output form (per-state scalar, progress delta, ranking preference, executable reward function).
The second layer opens the box and groups the ways rewards get constructed into four families. Frozen foundation-model scoring computes reward-like signals from pretrained models with no training, for example CLIP image-text similarity between the current frame and the language goal, or TOPReward asking a VLM "is the task done?" and taking the probability of the "true" token. Temporal and relative supervision learns from demonstration order (later frames are closer to done) or from preference comparisons between states. Instruction tuning treats progress prediction as explicit instruction following, with VLMs asked to reason before scoring (RoboReward, ProgressLM, Robo-Dopamine). Programmatic construction writes the reward as executable code (Text2Reward, Eureka), which is interpretable and editable.
The third layer ties methods to their data and evaluation, splitting evaluation into progress fidelity (calibration, temporal consistency, relative ordering, task grounding, uncertainty), robustness and generalization (cross-task, cross-view, cross-embodiment, non-monotonic execution), and downstream utility (online RL, offline learning, retrieval and filtering, planning).
The survey runs no new experiments; its output is the selection map itself. The main findings read as follows. Frozen-VLM scoring works zero-shot, but the score behaves more like a semantic prior than a calibrated reward and usually needs normalization, thresholding, or prompt engineering. Temporal and relative supervision is cheaper to label than precise scores, but comparisons have to be converted into reward functions and assume preferences can be represented by a consistent score. Instruction tuning is the most explicit family, baking progress prediction into the prompt and training target. Programmatic rewards are flexible and interpretable, at the cost of shifting the hard part from reward prediction to task decomposition and reliable state access, so the code can be correct yet still misaligned with intended behavior.
Data construction lines up by human involvement: fully manual captures semantics that are hard to quantify (grasp stability, acceptable contact) but is expensive; human-in-the-loop labels only keyframes and interpolates; fully automated uses temporal structure or foundation models to label at scale but assumes progress tracks time, or only reflects what state variables can express.
For anyone doing robot RL or reward engineering, the past workflow was to pick a method by name and then struggle to articulate its strengths and weaknesses. This map gives a reference frame: given a progress reward method, you can state its interface assumptions, which supervision family it belongs to, and whether its claimed gains were validated on fidelity or on downstream utility. The evaluation taxonomy is a ready-made checklist for anyone building a benchmark.
The survey names four shared weaknesses across the field. Estimation granularity is generally coarse; sparse sampling and discrete progress bins miss fine alignment, slight object motion, or early contact failure. The fixed-rate assumption is the biggest one: many methods use temporal order or normalized timestamps as supervision, which silently assumes progress is uniform, when real tasks are full of plateaus and sudden jumps. On latency, large VLMs need to read multiple frames, interpret instructions, and reason before scoring, fine for offline evaluation but infeasible at every control step. Long-horizon memory is the hard one: the same frame means different things at different stages (repeated grasps look identical but reflect different completion counts), and memoryless models score them the same.
These are field-level gaps, not flaws of the survey. One caveat to name: the survey is dense with method citations but gives almost no quantitative cross-method comparison, so the quantitative part of the selection map is largely left to the reader.