NVIDIA's Ming-Yu Liu Explains Cosmos 3: One World Model for Video, Audio and Robot Actions
Machine Learning Street Talk · rss · 2026-09-16
Machine Learning Street Talk interviews Ming-Yu Liu, who leads NVIDIA's Cosmos research, on Cosmos 3—the omnimodal world model that generated the never-filmed left-turning car opening the episode.
Highlights:
- Architecture: a vision language model reasons token by token; its weights initialize a bidirectional diffusion generator producing video, audio and robot actions, aligned by a shared temporal position scheme.
- World models as tools: forward dynamics, inverse dynamics and policy trained together under a capacity limit so each helps the others.
- Data: plentiful first-person human video transfers to data-starved robots; Cosmos post-trained on DROID is a solid starting point for pick-and-place policies.
- Testing: a neural simulator only needs to rank policies like the real world would, letting teams pick checkpoints for real trials; Cosmos Dreams applies this closed-loop idea to driving and robotics.
- Three sizes—Super, Nano, Edge (Edge targets Jetson Thor, Orin, DGX Spark)—with open weights, code and data. Episode is an NVIDIA paid partnership.
More from Embodied
- Wearing Whoop and Apple Watch on a single dual-slot band — tinyfool · 2026-11-02
- REA: async brain-cerebellum runtime for robots where policies never wait on the LLM — Pengfei Zhou · 2026-11-02
- Robot Combat Sports Coming to Japan with a Permanent Arena in Tokyo — cixliv · 2026-11-02
- REK robot fighting in the cage: humanoid bot in boxing gloves draws a crowd — broodsugar · 2026-11-02
- iPhone Duo: a folding iPhone mod impresses, but its hinge raises crackgate concerns — rudrank · 2026-11-02
- InflatableBots: University of Calgary's inflatable robots bring dynamic haptics to large-scale VR — lukas_m_ziegler · 2026-11-02