TaoMate: Alibaba & NJU Framework Solves Error Accumulation in Long Digital Human Videos
量子位 · wechat · 2026-08-02
Alibaba's Taotian Group and Nanjing University proposed TaoMate, an anchor-guided persistent memory framework for joint audio-video generation, aiming to solve recursive error accumulation and computational overhead in long digital human video generation.
Core Technical Approach
- Memory Separation & Anchor Constraints: Separates history into a bounded active context and a fixed-capacity persistent memory. It uses immutable visual anchors to constrain dynamic state updates, effectively suppressing color drift and appearance degradation.
- Decoupled Retrieval & Modulation: Uses residual side attention to retrieve audio-visual history and introduces Reference-AwareFiLM for channel-level appearance modulation, ensuring long-term consistency.
- Causal Distillation & Parallel Inference: Trains a student model with mixed rollout lengths to handle accumulated errors. It utilizes stage-parallel inference to achieve a 35.0 FPS full pipeline output on 3 GPUs.
Experimental Results
On a long-duration evaluation benchmark, TaoMate significantly improved LipSync and LongConsistency. Its DiT throughput reached 1.56x that of the next best method. The team also built a real-time interactive prototype supporting streaming generation, verifying its deployment potential for live digital humans.
More from Multimodal
- Valtz v0.1.0.11 adds FLUX.3 support, local image gen up to 4K on Mac — TgoAI · 2026-10-16
- H3 Character Swap LoRA V1.5 released with open ComfyUI workflow — linoy_tsaban · 2026-10-16
- ForcingBooster halves computation of autoregressive video diffusion with patch-wise denoising — VAST-AI · 2026-10-16
- Tokyo's Gemini AI Photo Booth turns visitors into plush toys and dioramas — bradneuberg · 2026-10-16
- AI face-swap of Seinfeld's George Costanza is uncannily convincing — alanskimp · 2026-10-16
- Orchestrating a 100-image portrait of humanity for aliens with ChatGPT, Argon and p5.brush — dx8xb · 2026-10-16