NVIDIA's Ming-Yu Liu on Cosmos 3: one model for video, understanding and robot actions
Machine Learning Street Talk · youtube · 2026-09-16
Machine Learning Street Talk interviews Ming-Yu Liu, who leads NVIDIA's Cosmos research. The left-turning car that opens the episode was never filmed — Cosmos 3 generated it. Liu explains how one model can describe video, generate it, and produce robot actions.
Architecture: a vision language model reasons one token at a time; its weights initialize a bidirectional diffusion generator for video, audio and action, with a shared temporal position scheme aligning signals running at different rates.
World models: Liu treats them as a toolkit — forward dynamics, inverse dynamics and policy trained together under a capacity limit so each helps the others.
Data transfer: plentiful first-person human video carries over to data-starved robots; a Cosmos model post-trained on DROID is a strong starting point for pick-and-place policies.
Testing (the most practical thread): a neural simulator doesn't need accurate success rates — it only needs to rank policies like the real world would, narrowing checkpoints for real trials. Cosmos Dreams applies this closed-loop idea to driving and robotics; Liu argues safety matters even more for humanoids around children and pets.
Sizes: Super, Nano and Edge (Edge targets Jetson Thor, Orin, DGX Spark), with open weights, code and data. Paid partnership with NVIDIA.
More from Embodied
- First M5 Ultra Geekbench leak: 52,516 multi-core, ~35% faster than M3 Ultra — jedisct1 · 2026-09-16
- Fei-Fei Li's World Labs is hiring robot learning researchers to scale world models — YunzhuLiYZ · 2026-09-16
- Figure CEO Brett Adcock Shows Off New F.03 Humanoid Robot at Dreamforce — adcock_brett · 2026-09-16
- Berseth: Self-driving shows cost cuts aren't enough — real-time closed-loop control is the hard part — GlenBerseth · 2026-09-16
- Light-Loco-Parkour shows humanoid robots zero-shot navigating complex terrain on onboard compute — chris_j_paxton · 2026-09-16
- Einride and Lidl put first cab-less Level 4 autonomous truck into daily operation on German public road — FlorianGallwitz · 2026-09-16