EvolvingAvatar: Interactive 3D Head Generation That Adapts as Conversations Unfold
Junjie Chen, Fei Wang, Kun Li, Yiqi Nie, Xun Yang, Yanbin Hao, Linfeng Zhang, Meng Wang
cs.CV
2026-09-29
A 3D chat avatar adapts via test-time training as a conversation unfolds, no motion labels; expression mismatch drops up to 11.1% on the hardest split, blind preference up to 90%.
An interactive 3D head has to speak and listen in the same stream: lips, expression, and neck pose aligned to its own audio while talking, nods and smiles while the other person talks. This line of work keeps widening the conditioning, from avatar audio only (DiffPoseTalk, ARTalk), to user audio (UniLS), to precomputed user FLAME motion on top (DualTalk). Once deployed, though, every one of these models runs with frozen parameters. The regularities inside a conversation, a user who expects nods, a pair's turn-taking rhythm, get consumed as context and never used as a learning signal. The paper's position: a live conversation is itself a stream of unlabeled audiovisual evidence, and the generator should learn while it generates.
EvolvingAvatar maps user face frames and both audio streams to FLAME parameters (106-D: 100 expression, 3 neck, 3 jaw), emitted every 200 ms. The core move is test-time training inside streaming generation, with four key choices:
The companion InterHead-Bench spans 455.95 hours and 4,365 participant IDs. A pipeline (DIALOG3D-FACTORY) unifies dual-view (Seamless Interaction) and single-view (RealTalk) recordings into aligned FLAME motion, speech, speaking states, and turn statistics; ID and OOD differ in whether participants and interactions are held out, and OOD-Hard switches to the single-view domain.
In parameter space, expression and neck FD (the motion analogue of FID, comparing generated versus recorded motion statistics) are the lowest of the five methods in both speaking and listening on every split. In blind A/B testing (10 raters, 60 trials), overall-realism preference over UniLS rises from 58% on ID to 66% on OOD and 72% on OOD-Hard; on OOD-Hard, preference reaches 90% against DiffPoseTalk and ARTalk and 86% against DualTalk.
Whether TTT actually learns shows in matched-checkpoint ablations (ID, expression P-FD, the joint user-avatar statistics distance):
| Ablation | Expression P-FD reduction from full TTT |
| No write (never update fast weights) | 20.4% |
| Freeze-1 (stop after first update) | 4.2% |
| No carry (clear fast state each interval) | 8.6% |
Over five equal-duration intervals, expression P-FD on OOD-Hard drops by up to 11.1% relative to the first interval (7.1% first to last) and by 2.7% on OOD, while all four baselines get worse on OOD over time.
The trade is explicit: DualTalk keeps lower MSE and lip/full-mesh error (LVE, MHD) with lower motion coverage (SID). The authors' reading is that these point metrics measure agreement with one recorded response, where the same context admits several plausible ones.
For teams building avatars for tutoring, companionship, or therapy support, this is a working demonstration that interactive motion generation can keep learning after deployment, per conversation, with zero labels. The model is small (159.57M total, 54.82M trainable), and its inputs are just camera frames and two audio streams, with no precomputed user FLAME trajectory. The generator trains for one epoch on 8 GPUs; streaming inference carries a 160 ms buffering delay, and the output can drive Gaussian-avatar rendering, robotic faces, and talking heads. The recipe of a label-free inner objective, two fast-weight lifetimes, and activity gating transfers directly to other streaming generation settings such as gestures, full-body motion, or robot policies.
The authors' own list: recorded conversations cannot capture how users react to generated motion, so whether the learned regularities hold in closed loop is untested, and the therapy-support use case needs clinician-guided studies. The numbers show cracks too. Full TTT worsens neck P-FD by 8.9% versus no updates, attributed to DCP not directly constraining per-region motion statistics. RGB visual input helps on OOD-Hard but hurts expression statistics on ID and OOD; the primary setting keeps RGB only to prove the end-to-end path, and the appendix shows that feeding pre-extracted FLAME scores better. The preference trend over UniLS is not statistically separable, with all three uncertainty intervals containing the 50% tie line. Module-level ablations (jaw head, activity, RGB) were run across model revisions and the authors label them exploratory; OOD-Hard covers a single recording domain.