Mirror learns Minecraft skills from video plans with no rewards or labels in ~30 minutes
vincesitzmann · x · 2026-10-20
Vince Sitzmann's team unveiled Mirror, a "learning-from-experience" framework where an agent learns to execute plans from a frozen video generative model. Two components only: a pretrained video planner and a video-to-action (inverse dynamics) model. The agent generates actions from the current V2A model, and each (video, executed action) pair becomes a valid training sample for self-bootstrapped retraining. Sample efficiency is remarkably high because the planner keeps pinning the agent on whatever critical action isn't working yet, focusing exploration. Results: navigation learned in 30 minutes of real gameplay, an iron pickaxe within 5 hours, even a diamond with an oracle planner — all without rewards, expert action labels, or any human-collected data, far outperforming RL in Minecraft. Led by Hyunwoo Ryu, Eric Chen, and Max Wilcoxson.
More from Embodied
- "Robots are going underground, figuring out their own routes" — a look past chatbot debates — VraserX · 2026-10-21
- Deepak Pathak: There Is No Perfect Data Source for Robotics — chris_j_paxton · 2026-10-21
- Discrete action representations match or beat continuous ones for robot policies — GuanyaShi · 2026-10-21
- Developer gave his Grok bot $1,000 to build itself a body — parts keep arriving — Baconbrix · 2026-10-21
- Tuesday Labs opens $200 reservations for 999 numbered Voxel Series-1 home robots — testingcatalog · 2026-10-21
- Tuesday Labs' home robot Voxel charms with Pixar-collaborated personality design — cneuralnetwork · 2026-10-20