JHU Study Compares Four Failure Data Sources for Robot Reward Models

A Johns Hopkins University team published the paper "The Right Kind of Wrong: Counterfactual Data for Reward Learning in Manipulation," tackling a practical question in robot RL: what kind of failure (counterfactual) data should be collected when training reward models for manipulation tasks?

Confirmed

The team systematically compared four sources of failure data—kinesthetic teaching (physical demonstration), policy rollouts, human video, and real2sim—evaluating their impact on reward model quality for robotic arm manipulation tasks
Results across three tasks show all four data sources improve average success rates
Costs differ sharply: kinesthetic teaching and teleoperation are expensive, while human video and real2sim are very cheap
Cheap sources (human video, real-to-sim) deliver notable gains at low data volumes
Scaling analysis shows adding failure data initially improves offline reward quality, but gains then plateau or even turn harmful—more failure data does not necessarily yield a better reward model

Why it matters

The study offers direct guidance on data collection strategies for robot reward models: rather than paying dearly for demonstration/teleoperation data, prioritize cheap sources like human video and real2sim, which deliver clear returns in the low-data regime
The "more failure data is better" intuition is disproven; the rise-then-fall scaling curve is a useful reference for both data budget planning and reward model training pipelines

2026-11-06 ~ 2026-11-06 · 7 related posts

Primary sources

Cheap failures work: human videos and real2sim rival kinesthetic teaching for robot reward models — mangahomanga · 2026-11-06 08:34 (UTC+08:00)
JHU study compares four failure data sources for training robot RL reward models — mangahomanga · 2026-11-06 08:34 (UTC+08:00)

Johns Hopkins study compares four failure data sources for robot reward models — mangahomanga · 2026-11-06
What failures should you collect? Comparing four data sources for robot reward models — mangahomanga · 2026-11-06
[source] Cheap failures work: human videos and real2sim rival kinesthetic teaching for robot reward models — mangahomanga · 2026-11-06
More failure data isn't always better: JHU finds reward quality plateaus then degrades — mangahomanga · 2026-11-06
Robotics study: low-fidelity sim matches high-fidelity with little data; more failures can hurt rewards — mangahomanga · 2026-11-06
How realistic must simulated failures be? Low fidelity suffices with little data — mangahomanga · 2026-11-06

1 near-duplicate retellings: mangahomanga