LiveFoley: real-time streaming video-to-audio generation with latency below perceptual thresholds
Yusheng Dai · hf · 2026-11-05
- A paper from Monash, Nanjing University, Tsinghua, Alibaba Token Hub and others, LiveFoley targets real-time streaming video-to-audio (V2A) generation for immersive interactive worlds (arXiv, Nov 4, 2025).
- Problem: existing streaming V2A shrinks visual frontend receptive fields to cut look-ahead latency, and causal diffusion backends denoise audio patches in isolation, hurting both alignment and quality. A user study sets perceptual delay thresholds: 0.34s for actions, 0.49s for speech, 0.85s for ambient.
- Method: a streaming causal autoregressive diffusion architecture using a Trailing Sliding Window for visual alignment and expanded temporal denoising windows for audio quality, with minimal latency.
- Ablations show the trailing design scores 0.61 DeSync at 0.08s look-ahead latency versus 0.88 for chunked (lower is better).
More from Multimodal
- Tencent Hunyuan Hy4 builds interactive 3D objects with working parts from a single prompt — mhdfaran · 2026-11-05
- Quixel makes Megascans free again and allows AI training on its 3D scan library — VraserX · 2026-11-05
- One prompt builds an interactive 3D alpine valley with Tencent Hunyuan Hy4 — no models, no textures, all code — Aiden_Tech_Ai · 2026-11-05
- A total beginner made an ad in under 10 minutes by pasting one link into Claude — PrajwalTomar_ · 2026-11-05
- 10 open-source GitHub repos for voice AI: cloning, TTS, and phone-call agents — victor_explore · 2026-11-05
- Generative world demo shows riverside scene with motorbike, bow and fishing rod — wavefnx · 2026-11-05