DeepMind Turns Video into Queryable 4D World

anselm · x · 2026-07-17

Google DeepMind showcased a system that transforms standard video walkthroughs into a "4D world queryable by natural language."

• The input is raw video, requiring no manual annotation or pre-labeling.
• The system breaks the video down into multi-layered representations: depth maps, surface normals, camera rays, and segmentation masks, reconstructing it into an interactive 4D scene.
• In the demo, users can directly ask:
• "I want to find a comfortable place to sit" — the sofa gets highlighted.
• "I need a whiteboard" — the whiteboard is located.
• "Is there a monitor I can use?" — the monitor is marked.
• "Where is my colleague?" — the system can spatially identify people.
• The implication: every video recorded in the past could become a searchable database, unlocking massive potential for robotics, AR, and spatial computing.

Original post →

More from Research

• Deep Learning Course Adds Mechanistic Interpretability Lecture This Winter — CSProfKGD · 2026-10-26
• INCLUDE-2.0 benchmark evaluates regional knowledge across 91 languages and 86 countries — ABosselut · 2026-10-26
• CoRL paper: flow-matching VLAs can run 1-step Euler inference with no retraining — kaixhin · 2026-10-26
• CoVer at EMNLP 2026: Training Code LLMs to Generate and Self-Verify Simultaneously — algo_diver · 2026-10-26
• Robotic dexterity should be judged on speed, force control and tool use, not copying humans — DJiafei · 2026-10-26
• Turing-complete malware can run in PyTorch's backward pass via a zero-valued loss — retr0jirachi · 2026-10-26

Research channel →