Kyutai's AURA streams talking-head avatars in real time on one GPU, 23× faster than its teacher

AURA: AUdio-dRiven streaming Avatar

Nathaniel Cohen, Nicolas Dufour, Amélie Royer, Alasdair Newson, Patrick Pérez

cs.CV, cs.AI

2026-11-20

Kyutai distills a bidirectional talking-head diffusion teacher into 4-step causal students: AURA 1.3B streams 576p in real time on one GPU, AURA 5B does 720p at 9.7 fps, 17× faster.

What problem this solves

Audio-driven diffusion models like TalkVerse, Wan-S2V and Hallo3 already synthesize convincing talking heads from a reference image plus a speech clip. The bottleneck is form, not quality. Bidirectional attention means they must buffer several seconds of audio and run dozens of denoising steps over a whole chunk before the first frame appears; on one GPU a multi-second chunk takes minutes, and generation leans on future audio. Voice assistants and live avatars need the opposite properties: render as audio arrives, hold memory flat, run forever. Prior streaming attempts (Talking Machines, Live Avatar, SoulX-FlashTalk) sit at 14-18B parameters and roughly 2 fps on a single GPU; Live Avatar needs a 5-GPU setup to reach 16 fps. A second obstacle is data: public talking-head clips are too short to cover a modern teacher's roughly 5 s generation window, so they cannot even seed distillation.

Method

AURA distills a bidirectional multi-step teacher into causal 4-step students that emit video chunk by chunk. The teacher is TalkVerse (5B, built on Wan2.2); for the 1.3B model the authors trained their own teacher on Wan2.1-1.3B with the same pipeline. Three changes on the student side:

That third choice is what keeps memory flat. Past chunks enter the cache only at the lowest noise level they reached, and one cache serves every denoising step, so memory is constant; schemes that cache multiple noise levels (Live Avatar) scale linearly with step count. The cache also rolls: the first chunk stays forever as an anchor, the same attention-sink trick streaming LLMs use, plus the single most recent chunk; everything else is evicted.

Training runs in two stages. Stage 1 is ODE teacher forcing: the teacher's 50-step sampling trajectories are precomputed and stored at five timesteps, and the student learns to predict clean latents directly from mixed-noise chunk sequences under an MSE loss. This bridges the architecture gap, since running DMD straight from teacher weights does not converge. Stage 2 applies DMD (Distribution Matching Distillation, matching the student's output distribution to the teacher's via reverse KL with an online critic) with backward simulation: the student first rolls out its own sampling trajectory without gradients, and the DMD loss is computed on those self-generated intermediate states, so training sees the inputs inference produces. The GAN loss from DMD2 is dropped; the authors cite its need for real video and unstable training. Training data is synthetic: 30K clips (38.7 h) generated by the TalkVerse teacher from Hallo3 reference images, LiveCC transcripts and TTS speech, released with the teacher's latent trajectories. Both students train as rank-128 LoRA; the 5B model used 64 H100s, with 120K steps for Stage 1 and only 5K for Stage 2.

Results

Main comparison on the Hallo3 test set, single H100 (the 1.3B model is evaluated at 480p/16 Hz here; native output is 576p):

MethodParamsResStepsTotal fpsFVD↓CSIM↑
Hallo314.5B480p500.22183.40.74
Wan-S2V14B720p400.08196.40.71
TalkVerse (teacher)5B720p500.58210.80.74
Live Avatar14B480p41.9181.10.77
AURA 5B5B720p49.7195.50.79
AURA 1.3B1.3B480p420.7177.10.84

Why it matters

Among the streaming talking-head systems compared here, AURA is the only one that is real time on a single GPU without a quality drop, and Kyutai releases everything: both students, the 1.3B teacher, and 30K clips with precomputed teacher trajectories. The 1.3B model already fronts Unmute, their cascaded voice assistant, running the full speech-conversation-plus-face loop on one H100. For avatar products this shifts the pipeline from generate-then-play to speak-and-render, with chunk-level latency around 0.64 s on the 5B model. Competitors either keep weights closed or need 4-5 GPUs. One honest caveat: the distillation spine is the CausVid/DMD2 line moved into the audio-driven setting; the new parts are the streaming audio conditioning, the constant-memory cache and the data work, not the distillation algorithm itself.

Limitations

The paper's own list: 5B identity drifts over minute-long generations (CSIM 0.76 to 0.69); the 1.3B model trails 5B and the teacher on visual quality, lip sync and naturalness, largely from lower resolution; and distribution metrics (FID/FVD/JEDi) miss local artifacts, since Hallo3 wins every distribution metric while showing exaggerated mouth motion and recurring hand artifacts, which is why the authors ran the human study. A close read adds more. The 1.3B teacher saw 31 h of real data against roughly 6,300 h for TalkVerse, so both its ceiling and the baselines it beats deserve skepticism. All timings are H100 numbers. The CelebV-HQ test audio is TTS-synthesized, so generalization to real recordings is not directly measured. And the concurrent Avatar Forcing uses a similar distillation recipe but needs an audio look-ahead window, where AURA claims to need only the current chunk; there is no head-to-head.

Terms

Source

Related papers

All paper explainers