JHU study compares four failure data sources for training robot RL reward models
mangahomanga · x · 2026-11-06
Johns Hopkins researchers released The Right Kind of Wrong: Counterfactual Data for Reward Learning in Manipulation, systematically comparing four sources of failure data for training reward models used in real-world robot RL.
Problem: RL beyond imitation needs rewards that capture task progress and failure, but robot datasets are dominated by successes and collecting real-hardware failures is expensive.
Four sources compared across five real manipulation tasks:
- Kinesthetic teaching (KT)
- Policy rollouts
- Human videos
- Real-to-sim rollouts
Setup: with tasks, success demos and evaluation fixed, failures are injected via LoRA adaptation of a reward model and few-shot prompting of a frozen model; evaluation covers trajectory ranking, partial-completion calibration and success detection on 250 human-annotated rollouts, plus RL post-training of π₀ to verify transfer. Paper, videos and code are public.
Related event: JHU Study Compares Four Failure Data Sources for Robot Reward Models(7 posts)→
More from Embodied
- Policy trained entirely in imagination matches PPO with 7.9× fewer simulator transitions — EpisodeYang · 2026-11-06
- Apple's mystery robot reportedly uses a Mac Mini as its planner brain — micoolcho · 2026-11-06
- Microdock v2 lets robot sit while docked, cutting servo power to zero — natesiggard · 2026-11-06
- Building an Asimov Robot in 22 Hours — and It Worked Out of the Box — whurley · 2026-11-06
- NEO-VLA drops the vision encoder: native VLA pretraining pays off on RoboTwin — liuziwei7 · 2026-11-06
- Gold Tesla Cybercab and Optimus shown at UP Summit — DMaguireARK · 2026-11-06