Google DeepMind on speech-to-speech: conversational, intelligent, multimodal — pick two

AI Engineer · youtube · 2026-09-15

Google DeepMind's Valeria Wu Fon and Tom Ouyang explain their speech-to-speech work in Gemini:

Pre-2018 speech recognition was a cascade of hand-built components; end-to-end models collapsed the chain but handled only one task.
Natively multimodal training on audio, video and text lets the model keep code-switched terms like "mid century" in a Spanish answer with no hand-written rules.
A three-way tension: raising the thinking budget lifts intelligence scores while time-to-first-audio falls apart.
Demos include multi-speaker live translation, a roadside-assistance agent, and proactive audio that ignores background noise.

More from Models