AnyTalk: Speech Animation for Arbitrary Characters Leveraging a Video Generation Model
Kwan Yun, Serin Yoon, Sunjin Jung, Jung Eun Yoo, Inyup Lee, Junyong Noh
cs.GR, cs.CV, cs.MM, cs.SD
2026-08-17
Zeroed-audio fine-tuning of a video diffusion model plus blendshape optimization gives any 3D character lip-synced speech with no animation data; LSE-D 11.3 vs ScanTalk 12.2.
Audio-driven 3D facial animation has an awkward prerequisite: most methods (CodeTalker, DiffSpeaker, FLAME-template pipelines) need paired audio-plus-3D data collected for the specific character, or require artists to re-rig the character onto a standard template. Production meshes vary wildly, so that premise breaks constantly. ScanTalk accepts arbitrary meshes but gives up precision and naturalness to do it.
AnyTalk poses a narrower question and gets a broader result: given any character that already has a blendshape rig, produce lip-synced speech animation with zero animation training data.
The move is to outsource motion knowledge to a video diffusion model. Hallo, an audio-driven portrait video generator, has learned from massive talking-head footage which mouth movements match an audio stream. Its outputs live in the photoreal-human domain, though, not on a stylized 3D render.
Character-specific Fine-tuning (CsF) closes that gap, and the zeroed audio embedding is the trick. The character is rendered once per blendshape, each still image is duplicated into a video, and the model is fine-tuned on these clips with the audio encoder's output zeroed. Audio drives motion in this architecture, so a zero embedding reads as "no motion": the model learns what the character looks like at rest while never seeing an animated frame. Temporal and attention layers stay frozen; only the spatial residual network trains, which preserves the pretrained motion prior.
At inference the audio is real speech and attention weights are re-set (pose 0, expression 1, lips 2), yielding a video where the head holds still and the mouth talks.
A second stage lifts the video back to 3D. Blendshape parameters B are optimized so that projected 3D landmarks match the 2D landmarks in the generated video, under three losses: talk-landmark alignment on expression-invariant landmarks with homography warping; an asymmetric mouth-opening loss weighted above 1, so the 3D mouth never opens less than the video; and an L1 regularizer keeping non-speech blendshapes at zero.
AnyTalkRT distills the pipeline into a small network using about 1,600 animations the full version generated, running at 9.09 ms per frame (110 FPS) in plain PyTorch.
Five test characters span 4,542 to 241,981 vertices and 32 to 121 blendshape parameters, stylized to photoreal; 150 animations over LibriSpeech audio.
| Method | LSE-D (lower better) | LSE-C (higher better) |
| AnyTalk | 11.304 | 3.155 |
| ScanTalk | 12.152 | 2.395 |
| DiffSpeaker+NFR | 13.857 | 0.665 |
| CodeTalker+NFR | 13.840 | 0.668 |
Against ScanTalk, the only other arbitrary-mesh baseline, both lip-sync metrics win by a modest margin. Against the two "+NFR" transfer baselines the gap is a blowout. A 21-person study (525 comparisons) prefers AnyTalk 78.6% on naturalness and 76.8% on lip-sync over ScanTalk, and 99.4% to 99.8% over the transfer baselines. AnyTalkRT lands at LSE-D 12.19 / LSE-C 2.96, trading accuracy for real time.
The cost model changes. Onboarding a new character used to mean capturing animation data or re-rigging; now it needs a blendshape rig and a set of renders. For small studios this turns talking-head animation from the most expensive part of the asset pipeline into an on-demand capability. The recipe also generalizes as a pattern: video diffusion as a motion prior, a zeroed conditioning embedding as the "still" signal, optimization to map 2D output back onto 3D parameters.
From the authors: about 3.12 seconds per frame in the optimization stage, so offline only; the rig is a hard prerequisite, and a character without a mouth-opening blendshape can fail; sparse 2D landmarks miss subtle expression detail and high-frequency motion; frontal view only, with multi-view generation lost to diffusion stochasticity; lip motion stays conservative relative to per-frame 2D methods, prioritizing 3D geometric stability.
Two more from reading the tables: the +NFR baselines are generic-face models pushed out of domain, so the 99%+ preference mostly measures their domain collapse rather than AnyTalk's absolute level; and audio coverage is LibriSpeech English only, with no singing, emotional speech, or other languages tested.