LiveFoley: Dynamic Real-Time Video-to-Audio Generation for Immersive Interactive Worlds
Yusheng Dai, Vladimir Iashin, Kangdi Wang, Yuxuan Jiang, Ilpo Viertola, Weiqiang Wang, Qiuhong Ke, Min Zhao, Jianfei Cai
cs.SD, cs.MM
2026-11-04
LiveFoley streams video-to-audio with a 0.08s lookahead, matching offline MMAudio-L on alignment (DeSync 0.48 vs 0.47) in 156 ms on one A800.
Interactive scenes need the sound to land with the picture. World models can emit video and audio together, but the audio is tied to their own video generator, so a game engine, an AR or VR runtime, or a live camera cannot feed them frames. Streaming video-to-audio (V2A) synthesizes sound from whatever pixels are arriving. Visual encoding and audio synthesis then share a delay budget that listeners do not notice.
A reference-free user study measured that budget. Discrete action sounds, such as impacts, become noticeable at about 0.34 s. Speech sits near 0.49 s. Ambient sound during free roaming is noticed only at about 0.85 s. A shooter and a stroll do not want the same latency.
Alignment in strong offline systems comes from a segment encoder, Synchformer: 16 frames, 0.64 s at 25 fps, with bidirectional attention inside the segment. A token exists only after the whole segment arrives, so V-AURA looks ahead 0.64 s. SoundReactor switches to per-frame DINOv2, looks ahead 0.08 s, and alignment drops. Shrinking the Synchformer window makes downstream alignment worse in step with the window, because alignment tracks how much time each segment covers. The audio path fails separately. Offline diffusion denoises every patch together. Streaming models use a local diffusion head (LocDiT) that denoises one patch at a time and sees history only through the language model hidden state.
LiveFoley is a sound engine hung off an existing visual pipeline. The frontend turns a frozen Synchformer into a streaming module. The backend widens the denoising window past a single patch.
The trailing sliding window pins the end of the segment on the current frame and feeds that frame plus history only. Missing history is the first frame repeated, so the first sound does not wait for 16 real frames. The receptive field stays eight tokens of bidirectional attention. Look-ahead follows the hop s: every 2s frames the window slides and emits s new tokens. Hops of 1, 2, and 4 mean 0.08, 0.16, and 0.32 s of frontend look-ahead. A shorter window throws away alignment. A shorter hop only runs the encoder more often.
Scenes that tolerate 0.34 s and scenes that tolerate 0.85 s should not each get an expert model. A two-layer Transformer, the Synchronizer, adds positions, aggregates the eight tokens, keeps the last s, and adds a learned latency-mode embedding. The three hops rotate across batches, and one checkpoint switches hop at inference. Every mode emits a 12.5 Hz token stream, so the backend stays the same. At s=4 the frontend runs a quarter as often as at s=1. That shared setup is the Unified Varhop framework.
Audio lives in a continuous latent from a causal VAE. The 16 kHz mono VAE runs at 25 Hz with two frames per patch. The 48 kHz stereo VAE runs at 50 Hz with four frames per patch. In both cases the language model steps at 12.5 Hz, one visual token per step. The LM is a 28-layer MiniCPM trained from scratch. LocEnc compresses the previous clean patch into an embedding and concatenates the current visual token; the LM turns that into a hidden state. A 12-layer LocDiT flow-matches this state from noise to a clean patch and hands it back. Flow matching trains the network to walk noise to data along a straight path.
History LocDiT is the default. The two previous clean patches enter the denoising window at no extra backend look-ahead. Those neighbors are already finished audio.
The next patch cannot simply join the window. Its condition depends on the clean current patch, which is still being denoised. That loop is the autoregressive deadlock. Future-Warmup builds an approximate condition from the current hidden state and the next visual token, then denoises the two patches together. Both patches sit in the training loss. At inference the future patch is only a warmup: it is dropped and regenerated at the next step with the true hidden state. The extra look-ahead is 0.08 s. Halving the autoregressive rate so one step covers two patches also sees the future, and alignment gets worse, so that variant is only a control.
Sequences longer than the roughly 9.92 s training length use an attention sink and a rolling KV cache.
The main table is the VGGSound test set, on 8-second clips. FDPANNs and FDPaSST are Fréchet distances to real audio, and lower is closer. IS and PQ score quality. ImageBind (IB) scores semantic fit. DeSync is the predicted offset in seconds, and lower is tighter. The default is History LocDiT at s=1, so the frontend looks ahead 0.08 s.
| Method | Metric | Result |
| MMAudio-L (offline) | DeSync / FDPANNs / FDPaSST / IS / IB | 0.47 / 8.48 / 74.16 / 11.62 / 0.31 |
| LiveFoley-48k | DeSync / IS / PQ | 0.48 / 11.42 / 6.26 |
| LiveFoley-16k | FDPANNs / FDPaSST / DeSync / KL / IB | 9.56 / 92.48 / 0.50 / 2.24 / 0.29 |
| WorldSonus (0.10 s) | DeSync / FDPANNs / FDPaSST | 0.67 / 16.32 / 197.68 |
| V-AURA (0.64 s) | DeSync / FDPaSST | 0.69 / 245.25 |
| SoundReactor (quoted, 0.08 s) | DeSync / FDPaSST | 1.03 / 113.00 |
LiveFoley-48k reaches DeSync 0.48, level with offline MMAudio-L at 0.47. Other streaming systems land between 0.67 and 1.03. LiveFoley-16k has the best streaming distribution match, FDPANNs 9.56 and FDPaSST 92.48. That 92.48 is still above MMAudio-L at 74.16, and higher is worse on this metric. The 48 kHz model posts the best PQ in the whole table, 6.26, while its FDPaSST rises to 128.34, behind the 16 kHz model. Offline AudioX scores IS 13.34 with DeSync 1.22. Alignment and quality do not crown the same system.
In listening tests LiveFoley leads the online systems, with MOS-Q 3.23 and MOS-A 3.58. Alignment approaches MMAudio-L at 3.79. A training-free chunked MMAudio records the best KL in the table, 2.11, and its seams are audible.
Frontend ablations use VGGSound-Curated, 1,485 clips with speech, faces, and off-screen sound removed, on a fixed V-AURA backbone. At 0.08 s of look-ahead, SigLIP2 and DINOv2 score DeSync 1.14 and 1.10. Trailing-window Synchformer scores 0.61. VideoMAE-V2 needs 0.64 s to reach 0.88. Cutting a chunked window from 0.32 s to 0.08 s moves DeSync from 0.67 to 0.88. The trailing window stays between 0.60 and 0.61.
The unified Varhop model matches single-hop experts on DeSync, both 0.47 at s=1, with a small loss on distribution distance. Removing the Synchronizer costs about 0.03 DeSync. On the same ablation, History LocDiT cuts FDPANNs from 21.08 to 17.55 against an independent head, while FDPaSST moves from 154.62 to 158.04. Future-Warmup reaches 16.88 and 143.92, with DeSync 0.46. The half-rate control is worse than an independent head: DeSync 0.54 and FDPaSST 180.52.
On one A800 at s=1, end-to-end latency from the first frame to the first playable waveform is 156 ms. Adding Future-Warmup's 0.08 s puts the figure at about 236 ms, under the roughly 0.34 s threshold for action sounds.
Games and headsets already have a visual pipeline. They need a sound engine that accepts those pixels and can spend a different delay in a firefight than on a stroll. The trailing window keeps a segment encoder receptive field on the order of 0.64 s, looks ahead only 0.08 s, and does not retrain the encoder. One checkpoint covers hops 1, 2, and 4. Roaming can take s=4 and run the frontend about four times less often.
What reaches the offline level is temporal alignment. Quality stays behind. A causal model has to commit to each sound from the frames seen so far. History LocDiT is the gain that adds no delay. Future-Warmup buys another step by paying 0.08 s and using an approximate future condition. The published wall clock is only the A800 at s=1.
Matching offline models that denoise a whole clip at once is still open. Future-Warmup both sees one extra visual token and uses an approximate condition, and nothing separates those two effects inside the FDPaSST drop from 158.04 to 143.92. The 236 ms figure adds 0.08 s to the measured 156 ms. There is no fresh wall-clock run of that configuration.
DeSync is computed by Synchformer, and the visual condition is a frozen Synchformer. The comparison with MMAudio still holds: both use segment-level sync features, and 0.48 versus 0.47 is a tie. Comparisons with DINOv2 and SigLIP2 on DeSync share the metric with the condition, so the gap can look larger than a listener would hear. MOS-A and ImageBind move in the same direction. The listening-test sample size is only in the appendix.
The main table and the frontend ablation are different sets. Full VGGSound contains a lot of off-screen narration and background music. Curated removes that material, and the frontend ablation sits on V-AURA, not on LiveFoley. The user study is reference-free, and its protocol is distinct from the broadcast lip-sync recommendation ITU-R BT.1359. Whether 0.34 s still holds for fast actions in a headset depends on stimuli the main text does not show. Training used about 2,000 hours (745K clips), mixing in-the-wild video with egocentric footage. Evaluation is 8-second VGGSound clips. Longer sessions rely on the attention sink, and no long-form numbers are reported.