The Missing Temporal Link: Temporal Context Routing for Script-Driven Audio-Video Generation
Yichen Liu, Quanwei Zhang, Haozhe Wang, Donghao Zhou, Xiaojie Li, Yang Shi, Jiaming Liu, Ruihua Huang, Yingtian Zou, Daquan Zhou
cs.MM, cs.CV
2026-09-02
TCR routes script clocks into LTX-2.3 cross-attention, cutting shot-boundary MAE from 1.11s to 0.042s and lifting Dialogue [email protected] from 28.3% to 84.1% on 200 scripts.
Joint audio-video generators can now keep lips on beat with speech. Scripted production wants something stricter: cuts and lines landing on a specified clock. In short drama and ads, a structured prompt already names when each shot should start and when each line should be spoken. Models still miss those times. Video and audio stay synced with each other, and both drift off the script timeline.
The gap is architectural. Current joint generators share a temporal axis between video and audio, but encode shot and dialogue clocks only as text. Video-only local prompting (Presto, ShotAdapter) can bind a caption to a visual span. It does not route the same clock through both towers.
TCR sits on LTX-2.3, a 22B joint generator. It changes only video-text and audio-text cross-attention. Self-attention and audio-video cross-attention stay as they are. The prompt follows the MTSS schema: Reference, Shot, Event (dialogue in these experiments), and Global. Numeric timerange fields are stripped from the text and compiled into a token-level timing map.
Each token inherits its parent interval. TCR turns that interval into a duration-normalized Gaussian bias on the cross-attention logits. The score is 0 at the interval center and −β/2 at either endpoint. β is fixed at 5, so an endpoint keeps about 8% of the center weight. Shots and lines can overlap; each prompt is routed on its own clock, so a line can keep speaking across a cut. TCR adds no trainable parameters. Only LoRA adapters are optimized under the original joint flow-matching loss.
Supervision comes from a coarse-to-fine pipeline on two short-drama collections, yielding 57,022 clips. Clip cuts sit in speech-free regions so internal shot transitions survive. Gemini writes the structured script with coarse timestamps. PySceneDetect then replaces shot bounds and WhisperX realigns dialogue at word level, rounded to a 0.1 s grid. The test set is 200 scripts (640 shots, 441 lines) with no shared source-media id or caption hash.
Against open-source joint generators, TCR cuts Shot Boundary MAE from LTX-2.3's 1.11 s to 0.042 s (96%). Shot IoU rises from 0.532 to 0.957; exact shot-count accuracy from 36.0% to 93.0%; Dialogue [email protected] from 28.3% to 84.1%. Imaging quality (0.7032), Sync-C (2.78), and WER (8.48%) match or beat the baselines. Lip-offset accuracy is 30.5%, essentially tied with LTX-2.3 at 31.2%.
| Method | Shot B-MAE (s) | [email protected] (%) | Shot Count Acc. (%) |
| Wan2.2 | 2.31 | — | 9.0 |
| OVI | 1.84 | 8.6 | 17.0 |
| JoyAI-Echo | 1.81 | 23.2 | 15.5 |
| LTX-2.3 | 1.11 | 28.3 | 36.0 |
| TCR | 0.042 | 84.1 | 93.0 |
On the same backbone, Gaussian Interval RoPE lands at 0.113 s MAE and a hard interval mask at 0.108 s. TCR is more than 60% lower. Training on coarse timestamps instead of refined ones sends MAE back to 0.375 s and [email protected] to 37.6%. Feeding shots only to video and lines only to audio hurts dialogue timing and AV sync. In a 28-person blinded study on 16 clips, TCR takes 72.3% of overall votes against LTX-2.3 and 83.9% against JoyAI-Echo, ties included.
Script-driven short video is currently stuck at "it generated, the cuts are late." TCR shows you do not need to retrain the 22B backbone. Lift the clock out of the text and add it as an attention bias, and shots and lines land on the second without a quality tax. For teams already on an LTX-class joint generator, this is a LoRA control patch, not a new generative family.
There is no dedicated limitations section. The test is 200 short-drama scripts, one sample each, no first-frame conditioning, and no stress test on long-form, multilingual dialogue, or non-speech events. β=5 is shared across every block and head. The data pipeline is a Gemini / cut-detector / WhisperX cascade; mismatches are dropped, with no reported coverage or bias. The user study is 16 cases and 28 raters. Most importantly, TCR assumes the script already carries precise timerange fields. Turning a literary screenplay into that clock is still an upstream LLM job.